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ABSTRACT 


The  Chief  of  Naval  Operations  defines  FORCEnet  as  the 
"operational  construction  and  architectural  framework  for 
Naval  Warfare  in  the  Information  Age  which  integrates 
warriors,  sensors,  networks,  command  and  control,  platforms 
and  weapons  into  a  networked,  distributed  combat  force, 
scalable  across  the  spectrum  of  conflict  from  seabed  to 
space  and  sea  to  land."  The  Trident  Warrior  experiments 
are  the  Navy's  premier  FORCEnet  Sea  Trial  experiments.  The 
purpose  of  the  Trident  Warrior  experiments  is  to  provide 
speed  to  capability  and  to  develop  supporting  tactics, 
techniques,  and  procedures.  Speed  to  capability  provides  a 
rapid  fielding  of  improved  capabilities  to  the  fleet.  The 
supporting  tactics,  techniques  and  procedures  optimize  the 
employment  of  the  new  technology. 

The  purpose  of  this  thesis  will  be  to  provide  a  basic 
overview  of  the  Trident  Warrior  Experimentation  Process. 
Through  a  step-by-step  analysis,  this  thesis  will  explain 
and  justify  the  individual  steps  that  comprise  a  successful 
experiment/ experimentation  campaign . 
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I. 


INTRODUCTION 


A.  SEA  POWER  21 

In  October  2002,  Admiral  Vernon  Clark,  the  Navy's 
twenty-seventh  Chief  of  Naval  Operations  (CNO) ,  laid  out 
his  vision  for  the  Navy  of  the  future  in  Sea  Power  21. 
Adm.  Clark  foresees,  "Future  naval  operations  [that]  will 
use  revolutionary  information  superiority  and  dispersed, 
networked  force  capabilities  to  deliver  unprecedented 
offensive  power,  defensive  assurance,  and  operational 
independence  to  Joint  Force  Commanders"  (Sea  Power  21)  . 
The  three  pillars  of  Sea  Power  21  are  Sea  Strike,  Sea 
Shield,  and  Sea  Basing. 


SEA  POWER  21 

Sea  Shield 


Sea  Basing 


Figure  1.  Sea  Power  21 (From:  Sea  Power  21) 

Sea  Strike  is  the  ability  to  inflict  a  persistent  and 
precise  offensive  force.  In  order  to  provide  this 

offensive  power.  Sea  Strike  must  be  capable  of  persistent 
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intelligence,  surveillance,  and  reconnaissance;  time 
sensitive  strikes;  electronic  warfare  and  information 
operations;  ship-to-ob j ective  maneuver;  and  covert  strike. 
Sea  Strike  will  provide  an  amplified,  ef f ects-based 
striking  power;  increase  effectiveness  of  precision 
attacks,  enhance  the  war  fighting  contribution  of  Marines 
and  Special  Forces,  and  allow  seamless  integration  of  joint 
strike  packages.  The  capabilities  associated  with  Sea 
Strike  will  continue  to  grow  with  technological  advances  in 
unmanned  combat  vehicles,  hypersonic  missiles,  electro¬ 
magnetic  rail  guns,  and  a  vast  array  of  sensor 
improvements . 

Sea  Shield  provides  defensive  assurance  to  our  forces 
and  our  allies  across  the  globe.  Sea  Shield  will  provide 
layered,  global  defensive  power  based  on  control  of  the 
seas,  a  forward  presence,  and  networked  intelligence,  in 
order  to  enhance  homeland  defense,  assure  littoral  access, 
and  project  power  inland.  Future  technologies  within  Sea 
Shield  include  integrating  existing  technology  into  a 
common  undersea  picture  and  a  single  integrated  air 
picture,  intelligence  and  communications  reach-back 
systems,  theater  missile  defense,  organic  mine 
countermeasures,  and  autonomous  unmanned  vehicles.  Ideally 
these  technologies  will  expand  combat  reach,  create  a 
common  operating  picture,  and  improve  self-defense 
capabilities  to  ensure  sea  superiority. 

Sea  Basing  provides  operational  independence  to  naval 
forces,  as  well  as  support  for  the  joint  force.  Sea  Basing 
will  provide  pre-positioned  war  fighting  capabilities, 
enhance  joint  support  for  a  dispersed  naval  force,  increase 
force  security  and  operational  agility,  and  minimize 
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reliance  on  shore  infrastructure.  Sea  Basing  must  provide 
positioning  afloat,  offensive  and  defensive  power 
projection,  logistical  and  command  and  control  (C2) 
support,  and  accelerated  deployment  and  employment 
timelines  for  joint  assets.  In  order  to  meet  these 
challenges.  Sea  Basing  will  need  increased  capabilities  to 
move  heavier  cargo  faster,  the  ability  to  access  all  cargo 
while  at  sea,  and  enhanced  and  integrated  Joint  C2  and 
logistics  capabilities. 

Sea  Power  21  implements  a  Global  Concept  of  Operations 
(GCO)  that  disperses  combat  power  through  a  variety  of 
platforms  that  possess  an  immense  warfighting  capability. 
This  will  allow  us  to  "dissuade,  deter,  and  defeat  both 
regional  adversaries  and  trans-national  threats"  (Sea  Power 
21)  . 

While  Sea  Strike,  Sea  Shield  and  Sea  Basing  are  the 
three  pillars  of  Sea  Power  21,  a  triad  of  organizational 
processes.  Sea  Warrior,  Sea  Enterprise,  and  Sea  Trial,  will 
be  used  to  align  and  accelerate  the  development  and 
incorporation  of  new  technology  for  the  fleet.  Sea  Warrior 
will  invest  in  the  Navy's  human  capital.  Today's  Navy 
employs  cutting  edge  technology,  requiring  naval  personnel 
to  be  both  highly  skilled  and  highly  trained.  Sea  Warrior 
will  provide  Sailors  who  are  skilled,  motivated,  and 
properly  employed,  producing  a  more  effective,  more 
efficient  fleet.  Sea  Enterprise  will  incorporate  modern 
business  practices  to  reduce  costs  and  improve  efficiency. 
Sea  Enterprise  seeks  to  further  improve  efficiency  by 
seeking  opportunities  to  work  with  other  services  on  common 
goals.  Joint  operations  save  money,  improve  inter¬ 
operability,  and  promote  system  integration. 
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The  final  process  designed  to  implement  Sea  Power  21 
more  smoothly  and  effectively  is  Sea  Trial.  Sea  Trial  is  a 
fleet-led  process  of  concept  and  technology  development 
that  delivers  enhanced  capabilities  to  the  fleet.  Sea 
Trial  will  coordinate  the  fleets,  technology  development 
centers,  and  academic  resources  to  integrate  wargaming, 
experimentation,  and  exercises  that  will  increase 
warfighting  capabilities. 

We  often  cite  asymmetric  challenges  when 
referring  to  enemy  threats,  virtually  assuming 
such  advantages  belong  only  to  our  adversaries. 

"Sea  Power  21"  is  built  on  a  foundation  of 
American  asymmetric  strengths  that  are  powerful 
and  uniquely  ours.  Among  others,  these  include 
the  expanding  power  of  computing,  systems 
integration,  a  thriving  industrial  base,  and  the 
extraordinary  capabilities  of  our  people,  whose 
innovative  nature  and  desire  to  excel  give  us  our 
greatest  competitive  advantage.  (Sea  Power  21) 

B .  FORCENET 

FORCEnet  is  the  method  in  which  the  U.S.  Navy  will 
build  on  its  asymmetrical  strengths.  FORCEnet  is  designed 
to  be  "the  operational  construct  and  architectural 
framework  for  naval  warfare  in  the  information  age, 
integrating  warriors,  sensors,  command  and  control, 
platforms,  and  weapons  into  a  networked,  distributed  combat 
force"  (FORCEnet:  A  Functional  Concept) .  FORCEnet  is  often 
described  as  the  'glue'  that  holds  Sea  Power  21  together. 
FORCEnet  will  improve  the  speed  and  accuracy  of  the 
decision-making  process,  while  integrating  knowledge  and 
providing  a  common  operating  picture  to  dominate  the 
battlespace.  This  will  ultimately  allow  commanders  to  make 
timelier,  better-informed,  and  more  accurate  decisions.  By 
increasing  accessibility  to  information,  FORCEnet  will 
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create  a  network  effect,  exponentially  increasing  the  value 
of  information  or  capabilities  provided  by  the  network. 
FORCEnet  will  provide  multi-tiered  weapons  grids; 
survivable  networks;  automated  decision  aids;  and 
distributed,  collaborative  C2,  while  maintaining  human¬ 
centric  integration.  FORCEnet  will  continue  to  promote  a 
highly  adaptive  and  decentralized  form  of  command  and 
control,  empowering  unit  commander's  initiative, 
adaptability  and  increased  tempo,  while  maintaining  the 
coordination  of  unity  of  effort  that  is  provided  by 
centralization. 

We  have  been  talking  about  network-centric 
warfare  for  a  decade,  and  FORCEnet  will  be  the 
Navy's  plan  to  make  it  an  operational  reality. 
Supported  by  FORCEnet,  Sea  Strike,  Sea  Shield, 
and  Sea  Basing  capabilities  will  be  deployed  by 
way  of  a  Global  Concept  of  Operations  that  widely 
distributes  the  firepower  of  the  fleet, 
strengthens  deterrence,  improves  crisis  response, 
and  positions  us  to  win  decisively  in  war.  (Sea 
Power  21) 

C .  TRIDENT  WARRIOR 

Trident  Warrior  (TW)  is  the  FORCEnet  component  to  Sea 
Trial.  It  is  an  annual  event  that  focuses  on  pairing 
concepts  and  technology  to  provide  enhanced  capabilities  to 
the  fleet  as  rapidly  as  possible.  The  Naval  Network 
Warfare  Command  (NETWARCOM)  is  the  Sea  Trial  Operational 
Agent  for  FORCEnet,  and  is  responsible  for  prioritization, 
coordination,  validation,  and  oversight  of  all  events 
within  Trident  Warrior  (TW)  .  NETWARCOM  is  the  operational 
agent  (OA) ,  and  the  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR) ,  is  the  chief  engineer  for  the  Trident  Warrior 
experiments.  The  experiments  seek  to  improve  FORCEnet 
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Command  and  Control  capabilities  by  providing  the 
warfighter  with  information  superiority  over  an  adversary, 
which  would  allow  the  warfighter  to  make  decisions  and 
execute  commands  more  efficiently.  By  providing  these 
capabilities,  along  with  full  supportability  and 
maintainability.  Trident  Warrior  underwrites  success  in  the 
battlespace . 

Trident  Warrior  seeks  to  provide  or  improve  upon  many 
of  the  aforementioned  capabilities  of  FORCEnet,  including: 
multi-tiered  sensor  and  weapon  information;  distributed, 
collaborative  command  and  control;  dynamic,  multi-path  and 
survivable  networks;  automated  decision  aids;  and  human¬ 
centric  integration.  Trident  Warrior  evaluates  new 
technologies  and  approaches  to  resolving  fleet  needs  using 
fleet  experimentation  and  exercises.  This  is  done  in  a 
spiral  development  process  that  enables  the  delivery  of  the 
new  capability  to  be  synchronized  with  the  delivery  of  the 
reguired  policies;  tactics,  techniques,  and  procedures 
(TTP) ;  and  concept  of  operations  (CONOPS)  .  Trident  Warrior 
measures  the  benefits  of  proposed  technology  and 
demonstrates  its  real  capabilities. 

The  end  state  of  a  Trident  Warrior  experiment  is  a 
Military  Utility  Assessment  (MUA)  that  will  provide  a 
recommendation  for  advancements  or  changes  in  Doctrine, 
Organization,  Training,  Materiel,  Leadership,  Personnel, 
and  Facilities  (DOTMLPF) . 

D.  FORCENET  INNOVATION  AND  RESEARCH  ENGINE 

In  order  to  facilitate  the  rapid  dissemination  of 
data,  the  Trident  Warrior  team  utilizes  the  FORCEnet 
Innovation  and  Research  Engine  (FIRE  website) . 
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"FIRE  was  developed  out  of  the  need  for 
structured  data  collection,  data  reconstruction 
for  analysis  and  generation  of  TW  analysis 
reports.  No  such  system  previously  existed,  and 
the  Naval  Postgraduate  School  (NPS)  the  analysis 
lead  for  TW,  was  asked  to  examine  different 
approaches.  NPS  developed  FIRE  as  an  enterprise 
computing  solution,  based  on  Oracle  9i  and  Oracle 
lOg  technology,  with  unique  artificial 
intelligence  (AI)  applications  included  in  the 
design."  (Poeltler  and  Gallup,  30) 

FIRE  has  become  a  very  useful  tool  for  the  Trident 
Warrior  team.  Access  to  FIRE  allows  the  team  to  share  both 
proprietary  and  confidential  information,  monitor  the 
status  of  major  objectives,  and  consult  major  reference 
publications  that  are  written  in  conjunction  with  the 
Trident  Warrior  experiments. 

E.  TRIDENT  WARRIOR  EXPERIMENTATION  PROCESS 

Every  Trident  Warrior  experiment  begins  with  an 
overarching  concept.  This  overarching  concept  is  the 
general  theme  of  the  experiment.  Recent  themes  of  Trident 
Warrior  Experiments  include  the  Global  War  on  Terrorism 
(GWOT)  and  the  Command  and  Control  (C2)  of  an  Expeditionary 
Strike  Group  (ESG) .  Several  key  initiatives  that  will  make 
significant  contributions  to  the  warfighter  within  this 
overarching  concept  are  then  identified.  Each  initiative 
is  then  dissected  into  multiple  objectives  that  are 
essential  to  initiative  development.  Goals  that  are  needed 
to  realize  the  objectives  are  identified.  The 
experimentation  team  then  describes  attributes  that  the 
proposed  FORCEnet  solution  will  need  to  meet  each  goal. 
Finally,  the  experiment  is  executed  to  compile  data  that 
will  validate  the  attributes. 
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Figure  2 .  Breakdown  of  an  Experiment 
In  order  to  promote  successful  experimentation. 
Trident  Warrior  has  established  the  Trident  Warrior 
Experimentation  Process.  By  following  the  steps  of  this 
process  in  order  and  within  the  prescribed  timeframe,  the 
Trident  Warrior  team  has  maximized  the  potential  for 
success  within  and  experimentation  campaign.  The  steps  of 
the  Trident  Warrior  Experimentation  Process  are: 

1.  Establish  Team 

2 .  Concept  Development 

3.  Technology/Tactics,  Techniques,  and  Procedures  (TTP) 
Harvest 

4.  Asset  Identification 

5.  Develop  Experiment  Objectives 

6.  Integrated  Definition  (IDEF)/  Operational  Sequence 
Diagrams  (OSD)  /Process  Action  Maps 

7.  Experiment  Design 

8.  Event  Definition 

9.  Data  Collection  Plan 
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10.  Execution 

11.  Final  Report 

12.  Assessment  Operational  Agent  Assessment  (OAA) 

13.  Military  Utility  Assessment  (MUA) . 
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II.  ESTABLISHING  A  TEAM 
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Figure  3 


Establish  Team 


An  experimentation  campaign  is  a  major  undertaking 
that  will  require  various  skills  and  expertise.  Many 
individuals  will  be  required  to  perform  the  numerous  tasks 
required  for  success.  Every  individual  will  bring  with 
them  their  own  skill  sets,  personality  traits,  and  ideas  on 
how  they  believe  the  experiment  should  be  run.  Balancing 
the  needs  and  desires  of  each  individual,  while  ensuring 
that  the  group  reaches  its  objective,  can  be  a  daunting 
task.  In  order  to  conduct  a  successful  experiment,  these 
individuals  must  refocus  and  realign  their  priorities  to 
become  a  team. 
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A.  WHAT  IS  A  TEAM? 

"A  team  is  a  small  number  of  people  with  complimentary 
skills  who  are  committed  to  a  common  purpose,  performance 
goals  and  an  approach  for  which  they  hold  themselves 
mutually  accountable"  (Katzenbach  and  Smith,  1994). 

1 .  Small  Numbers 

According  to  a  study  conducted  by  Katzenbach  and  Smith 
in  1994,  most  productive  teams  consist  of  a  small  number  of 
people,  usually  between  2  and  25  people.  The  most  highly 
effective  teams  usually  consisted  of  less  than  10  people. 
Teams  comprised  of  a  larger  number  of  people  experience 
difficulty  in  interacting  constructively  as  a  group.  Large 
teams  also  have  even  more  problems  trying  to  agree  on 
specific  steps  that  should  be  taken. 

2 .  Complimentary  Skills 

The  fundamental  tenant  of  teamwork  is  that  the 
combined  efforts  will  be  greater  than  the  sum  of  the 
individuals  working  alone.  In  order  to  maximize  the  team 
efforts,  it  is  necessary  that  teams  be  comprised  of  members 
with  varying  skills  sets.  In  general,  there  are  three 
types  of  basic  skills  sets:  technical  skills,  problem 
solving/decision-making  skills,  and  interpersonal  skills. 
A  technical  skill  is  a  functional  expertise  within  a 
specific  area,  such  as  marketing  or  engineering.  A  problem 
solving/decision-making  skill  can  be  used  to  identify 
problems  and  opportunities,  evaluate  options,  and  make  the 
necessary  trade-offs.  There  are  a  variety  of  interpersonal 
skills,  including  "risk  taking,  helpful  criticism, 
objectivity,  active  listening,  giving  the  benefit  of  the 
doubt,  support,  and  recognizing  the  interests  and 
achievement  of  others"  (Katzenbach  and  Smith,  1994) .  It  is 
important  to  note  that  both  problem  solving/decision-making 
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skills  and  interpersonal  skills  can  be  developed  and 
enhanced  throughout  the  life  of  the  team. 

3 .  Common  Purpose  and  Performance  Goals 

True  teams  can  be  identified  by  their  strong  desire  to 
achieve  some  greater  common  purpose.  A  common,  meaningful 
purpose  will  set  the  tone  and  direction  for  the  team.  The 
team' s  common  purpose  should  broadly  frame  the  requirements 
and  define  the  boundaries  and  scope  clearly  enough  to  keep 
the  team  on  track.  However,  it  must  also  be  flexible 
enough  to  allow  the  team  to  evolve.  This  broad  directive 
can  then  be  narrowed  into  specific  performance  goals.  By 
creating  specific  performance  goals,  the  team  will  be  able 
to  track  their  progress:  either  the  task  is  complete  or 
incomplete;  the  objective  was  met  or  not  met. 

4 .  Common  Approach 

By  uniting  in  a  common  approach,  all  team  members  stay 
on  track  and  their  combined  efforts  are  maximized.  A 
team's  common  approach  should  identify  what  each  team 
member's  job  will  be,  how  schedules  will  be  set  and  adhered 
to,  how  the  team  will  agree  upon  and  modify  decisions,  and 
how  to  determine  the  best  approach  for  getting  the  job 
done.  In  addition  to  these  steps,  most  teams  have  members 
who  will  evolve  into  the  roles  of:  "challenging, 
interpreting,  supporting,  integrating,  remembering,  and 
summarizing.  These  roles  will  help  promote  mutual  trust 
and  constructive  criticism,"  (Katzenbach  and  Smith,  1994) 
within  the  team. 

5 .  Mutual  Accountability 

In  order  to  be  truly  effective,  a  team  must  hold 
itself  accountable.  Forced  accountability  from  a 

supervisor  usually  works  best  with  individual  efforts,  but 
self-accountability  within  a  team  will  promote  commitment 
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and  trust.  Each  team  member  must  keep  in  mind  that  if  one 
person  fails,  the  team  fails. 

B.  TYPES  OF  TEAMS 

1.  Purpose  Classification 

One  way  to  classify  teams  is  by  their  purpose  within 
an  organization.  Three  common  teams  within  this  area 
include : 

a.  Problem  Solving  Teams 

Problem  solving  teams  are  usually  comprised  of 
several  individuals  from  different  departments  of  an 
organization.  This  type  of  team  meets  regularly  as  an  off¬ 
line  discussion  group  that  seeks  to  improve  quality, 
efficiency,  and/or  work  environment.  They  have  no  power  to 
reorganize  work  or  change  the  roles  of  workers  in  the 
production  process.  Suggestions  resulting  from  problem 
solving  teams  have  been  found  to  reduce  costs  and  improve 
product  quality,  but  due  to  their  lack  of  authority,  have 
had  little  effect  on  how  work  is  organized  or  managerial 
behavior . 

Jb.  Special-Purpose  Teams 

Special-purpose  teams  are  formed  to  complete  a 
specific  duty,  such  as  designing  and  introducing  work 
reforms,  meeting  with  suppliers  and  customers,  and  linking 
separate  functions.  Special  purpose  teams  work  on 

decisions  at  higher  levels,  creating  an  atmosphere  for 
quality  and  productivity  improvements. 

c.  Self -Managing  Teams 

Self-managing  teams  are  smaller  teams  that 

produce  an  entire  product  or  provide  an  entire  service. 

Team  members  master  multiple  tasks  and  may  even  rotate 

between  jobs  within  the  team.  These  teams  take  over  all 
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managerial  duties,  including  scheduling,  ordering  supplies, 
and  hiring.  Self-managing  teams  fundamentally  change  the 
way  work  is  organized,  giving  employees  more  control, 
eliminating  the  need  for  multiple  management  tiers,  and 
removing  bureaucratic  barriers. 

2.  Task  Oriented  Classification 

Teams  can  also  be  classified  by  evaluating  the  tasks 
they  are  organized  to  accomplish.  These  three  categories 
are  teams  that  recommend  things,  teams  that  make  or  do 
things,  and  teams  that  run  things  (Katzenbach  and  Smith, 
1994)  . 

a.  Teams  that  Recommend  Things 

These  teams  include  task  forces,  project  groups, 
and  audit,  quality,  or  safety  groups  that  are  asked  to 
study  and  solve  particular  problems.  They  usually  have  a 
set  completion  date  and  would  benefit  greatly  from  a  fast 
and  productive  start.  Teams  that  recommend  things  should 
also  go  to  great  lengths  to  gain  commitment  from  the  people 
who  will  implement  their  recommendations. 

Jb.  Teams  that  Make  or  Do  Things 

Teams  that  make  or  do  things  are  performance 
driven  and  should  focus  on  critical  delivery  points. 
Critical  delivery  points  are  the  places  in  the  organization 
that  most  directly  influence  the  cost  and  value  of  the 
product . 

c.  Teams  that  Run  Things 

Teams  that  run  things  are  extremely  rare.  Often 
groups  of  executives  are  referred  to  as  a  team,  but  they  do 
not  meet  the  fundamental  definition  of  a  team.  Often, 
because  of  destructive  conflict  and  a  lack  of  trust,  teams 
at  the  top  are  difficult  to  create.  Although  not  as 
efficient  as  a  team,  a  working  group  is  more  easily 
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assembled  at  this  level.  A  working  group  presents  fewer 
risks  and  allows  its  members  to  maintain  an  individual 
focus . 

C .  TEAM  LEADERS 

A  good  team  leader  has  the  ability  to  shape  a  team' s 
vision  and  provide  necessary  guidance  and  oversight.  A  bad 
team  leader  can  disintegrate  a  team  into  a  group  of 
unfocused  individuals  that  can  actually  decrease 
productiveness  from  the  start.  When  forming  a  team  there 
are  five  critical  steps  involved  (Katzenbach  and  Smith, 
1994)  : 

1.  Establish  urgency  and  direction 

3.  Set  clear  rules  of  behavior, 

4.  Set  and  seize  upon  performance  oriented  tasks  and 
goals , 

5.  Challenge  the  group  regularly  with  fresh 
information,  and 

6.  Spend  lots  of  time  together. 

Establishing  urgency  and  direction  will  set  the  team 
down  the  correct  path  and  will  prevent  wasting  time  trying 
to  get  the  team  on  track.  It  is  important  to  pay  attention 
to  the  first  meeting  and  actions,  as  this  will  set  initial 
impressions.  If  team  members  see  a  hard  working,  focused 
leader,  it  is  likely  that  they  will  be  more  focused 
themselves.  Clear  rules  of  behavior  should  be  set 

regarding  attendance,  discussions,  confidentiality,  and 
expected  contributions.  Establishing  a  few  performance- 
oriented  goals  will  give  the  team  confidence  and  motivation 
to  begin  moving  forward  quickly.  Significant  challenges 
will  unify  the  team,  and  when  accomplished  will  provide  the 
team  with  determination  and  enthusiasm.  Finally,  spending 
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time  together,  both  scheduled  and  unscheduled,  will  help 
build  trust  and  reliability  between  team  members. 

A  good  team  leader  will  put  team  performance  first  and 
recognize  that  they  need  the  team's  help  to  accomplish  the 
goals.  A  team  leader  must  delicately  balance  between 
providing  too  much  command  or  too  little  guidance,  as  both 
can  stifle  a  team's  motivation.  A  team  leader  should 
follow  these  five  elements  for  good  team  leadership 
(Katzenbach  and  Smith,  1993) : 

•  Keep  purpose,  goals,  and  approach  relevant  and 

meaningful 

•  Build  commitment  and  confidence 

•  Manage  relationships  with  outsiders  and  remove 
obstacles 

•  Create  opportunities  for  team  members 

•  Do  real  work. 

Additionally,  a  team  leader  should  avoid: 

•  Blaming  people  or  allowing  individuals  to  fail 

•  Excusing  away  shortfalls  in  team  performance. 

D.  CONCLUSION 

"The  single  most  important  consideration  for  those 
responsible  for  experimentation  design  is  to  ensure  that 
current  expertise  is  available  to  support  the  plan"  (Code 
Of  Best  Practices-Experimentation) . 

The  potential  of  a  team  greatly  exceeds  the  potential 
of  the  individuals  within  that  team.  In  an  experiment, 
this  potential  must  be  realized  for  success.  The  Trident 
Warrior  ' 05  team  provides  an  excellent  blueprint  for 
modeling  other  experimentation  teams.  Every  individual  has 
a  particular  area  of  responsibility.  In  this  manner,  the 
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Figure  5.  Concept  Development 

A.  CONCEPT  DEVELOPMENT  CONFERENCE  (CDC) 


The  primary  event  of  the  concept  development  phase  is 
the  Concept  Development  Conference  (CDC) .  The  purpose  of 
the  CDC  is  to  define  the  experiment's  theme,  goals,  and 
focus  areas.  The  CDC  is  an  attempt  to  get  the  primary 
stakeholders  to  agree  on  the  overarching  concept,  or  theme, 
for  the  experiment.  Examples  of  overarching  themes  include 
the  Global  War  on  Terrorism,  Expeditionary  Strike  Group 
Command  and  Control,  and  Bandwidth  Management. 


B.  INPUTS  FOR  THE  CONCEPT  DEVELOPMENT  CONFERENCE 

Once  the  team  has  agreed  upon  an  overarching  concept, 
the  major  stakeholders  in  the  experiment  are  each  given  a 
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turn  to  present  possible  initiatives  for  the  experiment. 
They  also  relay  relevant  ideas,  along  with  thoughts  on  how 
to  improve  the  experiment.  Within  the  Trident  Warrior 
experiments,  each  stakeholder  is  given  an  opportunity  to 
briefly  define  their  proposed  initiative  and  link  their 
ideas  to  Sea  Power  21,  FORCEnet,  or  other  applicable 
military  doctrine.  By  the  end  of  the  CDC,  everyone  should 
have  a  common  understanding  of  the  overarching  focus  area, 
and  the  experimentation  team  can  begin  to  determine  which 
initiatives  they  will  investigate. 

When  presenting  a  proposed  initiative,  the  major 
stakeholders  should  concentrate  on  these  key  issues. 

1 .  Capability  Gap 

The  fundamental  purpose  of  military  experimentation 
should  be  to  provide  solutions  that  support  the  actual 
warfighter.  These  solutions  may  focus  on  changes  to 
current  Doctrine,  Organization,  Technology,  Materiel, 
Leadership,  Personnel,  and/or  Facilities  (DOTMLPF) .  In 
order  to  be  considered  for  military  experimentation, 
initiatives  should  attempt  to  fill  a  void,  or  "gap"  in 
current  abilities  that  would  allow  our  forces  to  achieve 
some  desired  future  or  current  capabilities.  As  stated  in 
the  CDC  for  TW  '06,  "We  are  not  doing  experimentation  for 
experimentation's  sake;  [our  experiments]  must  fill  gaps." 

2.  Current  Efforts/General  Info 

Military  experimentation  should  not  attempt  to 
recreate  knowledge  that  has  already  been  discovered. 
Initiatives  should  be  researched,  and  the  experimentation 
team  should  be  informed  of  previous  efforts  and  current 
developments  within  each  field.  A  general  overview  of 
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previous/ongoing  efforts  related  to  these  proposed  focus 
areas  should  be  outlined. 

3 .  Experiment  Relevance 

In  order  to  keep  the  experiment  to  a  manageable 
volume,  the  CDC  will  attempt  to  determine  if  an  initiative 
aligns  with  the  current  theme  of  the  experiment.  Again, 
teams  should  have  some  purpose  behind  their  experiments, 
and  not  run  experiments  for  the  sake  of  experimentation. 
In  military  experimentation,  this  purpose  is  to  fill  gaps 
in  warfighting  capabilities.  Allowing  too  many  unrelated 
initiatives  will  result  in  the  experiment  wandering  away 
from  its  overarching  concept  and  becoming  too  large  and 
unmanageable.  This  wandering  is  referred  to  as  scope 
creep.  According  to  NETWARCOM' s  Ron  Adamo  of  FORCEnet 
Experimentation-Future  Plans, 

Scope  creep  is  an  experiment's  most  dangerous 
threat.  A  solid  CDC  will  help  by  firmly  focusing 
the  experimentation  team  (and  stakeholders)  on 
the  key  objectives.  These  objectives  can  change 
during  the  rest  of  the  process  but  only  with 
everyone's  buy-in.  Then  possible  deviations  from 
the  original  plan  can  be  addressed  with  a  full 
understanding  of  how  they  affect  the  overall  plan 
and  available  resources. 

C.  CONCLUSION 

Once  the  team  has  agreed  on  a  major  focus  area  and 
relevant  initiatives,  they  should  review  their  roles  and 
responsibilities  within  the  new  initiatives.  This  will 
ensure  that  team  members  understand  the  part  they  will  play 
through  the  rest  of  the  experiment. 

The  output  of  the  CDC  should  be  a  concentrated  list  of 
initiatives  that  enables  each  of  the  experiment  leads  to 
begin  concentrating  their  efforts.  If  everyone  is 
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comfortable  with  this  level  of  focus  and  direction  at  the 
end  of  the  CDC  then  the  experiment  design  team  should  be 
able  to  get  to  work  in  developing  the  detailed  experimental 
objectives,  questions  and  metrics. 
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Figure  6.  Technology/TTP  Harvest 


A.  BROAD  AGENCY  ANNOUNCEMENT 

In  order  to  attract  innovative  ideas  both  from 
industry  and  from  other  areas  of  the  military,  it  may  be 
necessary  to  issue  a  Broad  Agency  Announcement  (BAA) .  A 
BAA  is  a  notice  from  the  government  that  requests 
scientific  or  research  proposals  from  private  firms 
concerning  certain  areas  of  interest  to  the  government.  The 
submitted  proposals  may  lead  to  future  contracts. 
According  to  the  Federal  Acquisition  Requirement-Part  35: 

BAA's  may  be  used  by  agencies  to  fulfill  their 
requirements  for  scientific  study  and 
experimentation  directed  toward  advancing  the 
state-of-the-art  or  increasing  knowledge  or 


23 


understanding  rather  than  focusing  on  a  specific 
system  or  hardware  solution.  The  BAA  technigue 
shall  only  be  used  when  meaningful  proposals  with 
varying  technical/scientific  approaches  can  be 
reasonably  anticipated. 

If  not  using  a  BAA,  some  other  method  of  attracting 
outside  technologies  should  be  employed.  These  methods 
should  describe  the  experiment's  broadly  defined  areas  of 
interest,  the  criteria  for  selecting  the  proposals,  specify 
the  period  of  time  during  which  proposals  will  be  accepted, 
and  contain  instructions  for  the  preparation  and  submission 
of  proposals. 

BAA' s  imply  program  funding  is  available  for  an 
eventual  buy  decision.  NETWARCOM  does  not  control  such 
funding,  so  TW  does  not  typically  issue  a  BAA.  SPAWAR 
provides  technologies  in  support  of  FORCEnet,  to  include  TW 
experimentation.  Additionally,  word  of  mouth  among 
industry,  academia,  and  other  military  agencies  attracts 
sufficient  interest  from  outside  sources. 

B.  ENTERPRISE  DATABASE  FOR  INNOVATIVE  SOLUTIONS 
OBSERVATION  NETWORK 

In  order  to  gather  beneficial  technology/tactics , 
techniques  and  procedures  (TTP)  more  efficiently,  the  TW 
experimentation  team  developed  the  Enterprise  Database  for 
Innovative  Solutions  Observation  Network  (EDISON) ,  a  web- 
based  portal  through  which  they  can  collect  and  review 
contractor-submitted  proposals.  EDISON  provides  a  single 
point  of  entry  for  all  information  technology  (IT), 
information  management  (IM),  and  human  systems  integration 
(HIS)  solutions  in  support  of  FORCEnet.  Additionally, 
EDISON  provides  a  powerful  data-mining  portal  capable  of 
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extracting  essential  information  from  existing  databases  to 
provide  the  experimentation  team  with  a  wide  awareness  of 
programs  and  innovative  solutions.  This  single  entry  point 
for  submission  of  FORCEnet  innovative  solutions  from 
industry,  academia,  the  Department  of  Defense  (DoD) ,  and 
the  Navy/Marine  Corps  team  helps  to  eliminate  the  costly 
development  of  redundant  capabilities  and  screen  out 
solutions  that  are  not  in  accordance  with  the  enterprise 
architectural  vision.  EDISON  provides  tracking  of  these 
innovative  solutions  from  discovery  to  acguisition, 
enabling  the  selection  of  optimal  ideas  from  all  sources. 
EDISON  required  that  the  data  only  be  entered  a  single  time 
and  maintains  the  security  of  proprietary  information 
(EDISON  website  within  FIRE) . 

C.  TECHNOLOGY  DEVELOPMENT  CONFERENCE 

After  all  proposals  are  collected  through  EDISON,  the 
Technology  Development  Conference  (TDC)  is  convened  to 
determine  which  proposals  will  be  accepted  and  incorporated 
into  the  experiment.  The  TDC  board  should  be  comprised  of 
decision  makers  from  the  command  that  is  sponsoring  the 
experiment  (NETWARCOM  for  TW)  ,  as  well  as  representatives 
from  the  Fleet  Sponsor.  The  first  act  of  the  TDC  board  is 
to  throw  out  all  proposals  that  are  not  relevant  to  the 
current  experiment,  or  are  not  in  line  with  current  Navy 
doctrine,  organizational  structures,  or  the  current 
political  environment.  The  Trident  Warrior  experiments  do 
not  seek  to  evaluate  proposals  for  changes  in  established 
doctrine  at  the  TDC.  These  proposals  must  be  routed 
through  the  Naval  Warfare  Development  Command  (NWDC) . 
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Proposed  doctrine  changes  can  then  be  forwarded  to 

NETWARCOM  for  experimentation. 

The  TDC  board  should  also  remove  any  proposals  that 
are  not  technically  feasible  or  will  not  fit  into  the 
agreed  upon  timeline  for  the  experiment.  The  Trident 
Warrior  experiments  have  found  it  useful  to  divide  the  rest 
of  the  TDC  into  2  phases.  During  the  first  phase, 
interested  government  entities  are  permitted  an  opportunity 
to  give  a  brief  presentation  on  technologies  they  wish  to 
include  in  the  experiment.  Other  government  entities  may 
want  to  include  their  technology  for  the  purpose  of  testing 
or  demonstrating  capabilities.  During  the  second  phase 
contractors  are  permitted  to  give  a  brief  presentation  on 
technology  they  feel  would  be  beneficial  to  include  in  the 
experimentation  campaign.  During  this  phase,  the  only 
parties  present  are  the  TDC  board  and  the  contractors 
giving  the  presentation.  This  separation  is  intended  to 
protect  contractor's  proprietary  information. 

In  order  to  be  approved  by  the  Technology  Development 
Conference,  a  proposal  must:  (1)  be  a  FORCEnet  innovative 

solution  adding  value  to  the  warfighter  by  enhancing 
current  capabilities  without  detracting  from  current 
readiness  or  capabilities;  (2)  be  aligned  within  the 
defined  high  level  FORCEnet  capabilities  requirements,  and 
concepts;  (3)  be  consistent  with,  and  build  up,  fleet 
priorities;  and  (4)  provide  feedback  to  the 

requirements/acquisition  process  for  Navy  wide  delivery  of 
FORCEnet  capability.  Additionally,  within  Trident  Warrior, 
The  objective  is  usually  to  leave  the  technology  onboard 
the  ships  provided  by  the  fleet  sponsor,  so  the  proposal 
should  attempt  to  deliver  end-to-end,  supportable  and 
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deployable  leave  behind  capabilities.  The  amount  of 
funding  required  from  the  experiment's  sponsoring  command 
(NETWARCOM  for  TW)  would  factor  into  the  decision  as  well. 
(EDISON  website  within  FIRE) . 

D.  CONCLUSION 

Technology  and  accepted  tactics,  techniques  and 
procedures  (TTP)  should  be  gathered  in  an  effort  to 
optimize  the  experimentation  effort.  The  level  that  these 
gathered  technologies  play  within  the  experiment  is 
dependent  on  the  experiment's  overarching  focus  area  and 
the  initiatives  of  the  experiment.  Regardless,  the 

gathering  of  technology  and  TTP  is  a  crucial  step  in  the 
experimentation  process.  This  step  will  ensure  that  the 
experimentation  team  fully  understands  the  technologies, 
systems,  tactics,  techniques  and  procedures  (TTP)  they  are 
about  to  test,  and  that  the  team  can  present  new  and 
innovative  methods  of  employing  these  technologies  and 
systems . 
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V.  ASSET  IDENTIFICATION  AND  EXPERIMENT  OBJECTIVES 
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Figure  7 


Asset  Identification  and  Experiment  Objectives 


A.  ASSET  IDENTIFICATION 

It  is  very  important  to  get  the  entire  team  to  accept 
the  experimentation  campaign  from  the  very  beginning.  An 
established  commitment  from  every  team  member  will  promote 
trust  and  confidence  within  the  team.  Military 

experimentation  has  an  added  challenge,  in  that  the 
platforms  and  some  of  the  assets  required  to  perform  the 
experiment  do  not  belong  to  the  experimentation  team. 
Military  equipment  and  platforms  are  produced  to  support 
and  defend  the  Constitution  of  the  United  States  in  the 
most  effective  manner  possible.  Sometimes  this  is  means 
fighting  wars,  and  sometimes  this  means  enhancing 
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warfighter  capabilities  through  experimentation.  Due  to 
the  volatility  of  world  affairs,  it  is  often  difficult  to 
get  a  solid  commitment  from  the  military  assets  needed  to 
perform  the  experiments.  To  make  the  situation  worse, 
military  experimentation  often  requires  that  experimental 
systems  be  installed  on  military  platforms,  requiring  an 
even  greater  commitment. 

1 .  Require  a  Commitment 

A  firm  commitment  must  be  obtained  from  the  commanders 
of  all  required  military  assets.  Without  the  confidence 
that  all  committed  assets  will  be  available  at  the  time  of 
experimentation,  planners  cannot  begin  to  develop  the 
experiment's  objectives.  Installation  of  experimental 

systems  should  be  scheduled  for  the  committed  systems  and 
platforms.  Scheduled  installation  could  take  as  long  as 
two  calendar  years  to  complete.  The  commitment  required 
from  these  commanders  is  very  burdensome,  but  also  very 
necessary . 

2.  Get  Buy-In 

The  experimentation  team  should  remember  who  has  the 
ultimate  authority  over  the  assets.  The  most  important 
support  to  acquire  is  the  individual  with  the  most 
influence  over  the  assets.  If  you  have  the  support  of  the 
boss,  the  staff  will  follow.  In  order  to  obtain  this 
support,  the  experimentation  team  should  be  prepared  to 
defend  their  efforts  and  detail  the  purpose  of  their 
campaign . 

3 .  Be  Flexible 

When  circumstances  begin  to  change,  it  is  important 
that  the  experimentation  team  remain  flexible.  World 
events  frequently  dictate  the  redeployment  of  assets. 
Platforms  and  systems  that  were  reserved  to  carry  out  the 
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experiment  may  no  longer  be  available.  New  resources  may 
be  available  in  their  place,  or  a  reduced  quantity  may  have 
to  suffice.  It  is  important  to  maintain  a  flexible 
timeline  and  an  adaptable  experimentation  plan  in  order  to 
overcome  these  last  minute  changes.  An  experiment  that 
improves  knowledge  is  valuable,  even  if  the  knowledge 
gained  is  less  than  expected. 

B.  DEVELOP  EXPERIMENT  OBJECTIVES 

Within  this  stage  of  the  Trident  Warrior  process,  the 
experimentation  team  attempts  to  refine  the  major 
initiatives  agreed  upon  in  the  Concept  Development 
Conference.  At  the  CDC,  the  team  agrees  upon  an 
overarching  theme  for  the  experiment,  as  well  as  some  high 
level  initiatives  or  focus  areas.  The  team  then  splits 
these  focus  areas  into  specific  objectives  to  accomplish, 
and  formulates  questions  within  each  objective.  Objectives 
relate  to  a  specific  capability  that  is  to  be  explored 
through  experimentation. 

Developing  objectives  is  fundamentally  a  social 
process  of  forming  a  shared  understanding  in  order  to 
determine  what  is  to  be  addressed  by  the  assessment. 
During  a  C2  assessment  problem  formulation,  the  analytic 
problem  is  decomposed  into  appropriate  dimensions  such  as: 
functions,  mission  areas,  structures,  command  echelons,  and 
C2  systems. 

Problem  formulation  is  an  iterative  process  that 
evolves  over  the  course  of  the  study.  [This] 
phase  should  identify  the  context  of  the  study 
(i.e.  actors;  threats;  relevant  previous  studies; 
and  political,  social,  historical,  economic, 
geographic,  technological  environments)  and 
aspects  of  the  problem-related  issues  (i.e. 
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issues  to  be  addressed,  assumptions,  independent 
variables,  and  constraints)  (COPB  for  C2 
Assessment ,  54)  . 

TW  objectives  are  broken  down  into  exceptional  detail 
by  decomposing  each  into  the  following  categories  for  each 
initiative : 

•  Initiative  Statement  (a  high-level  description  of  the 
initiative's  purpose) 

•  Objective  (operational  or  technical  capability  to  be 
produced 

•  Military  Utility  Assessment  (MUA)  Recommendation 
(recommendation  that  will  be  made  to  Military  Utility 
Assessment  Board  if  the  objective  is  a  success. 

•  Questions/Proposal  (Specific  aspect  of  the  initiative 
to  be  addressed. 

•  Operational  conditions  required  to  produce  valid  data 
relevant  to  the  question  being  asked 

•  Systems  conditions  required 

•  Information  conditions  required 

•  Attributes  and  measures  that  will  be  evaluated 

•  The  data  required  to  produce  the  assessment,  which 
will  produce  the  measures. 

This  step  can  take  several  months  to  complete  because 
a  typical  TW  can  generate  up  to  150  separate  initiatives. 
But  when  these  questions  are  focused  at  the  right  level  of 
detail,  the  rest  of  the  event  design  is  expedited. 

A  formal  statement  of  the  objective  can  be  used  to 
make  the  experiment  more  productive.  "The  statement  of  the 
[objective]  should  be  formulated  so  as  to  specify  the  whole 
issue  under  study,  not  just  the  specific  hypothesis  under 
detailed  analysis"  (COBP-Experimentation,  129)  .  This  high 
level  description  will  help  the  team  identify  all  the 
elements  within  the  objective  and  ensure  that  proper 
controls  are  in  place.  The  COBP  for  Experimentation 
suggests  that  when  writing  an  objective  statement,  it  is 
important  to  ensure  that,  regardless  of  the  result  of  the 
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experiment,  the  overall  knowledge  of  the  event  is 
increased.  This  would  require  that  the  statement  be 
written  in  such  a  way  that  it  compares  two  outcomes.  This 
typically  requires  the  objective  to  establish  a  base  line, 
or  placebo,  with  which  to  compare  the  test  case. 
Unfortunately  real  world  experimentation  is  usually  unable 
to  create  two  identical  situations  in  which  to  compare  test 
and  baseline  results.  The  Trident  Warrior  experiments 
consider  current  systems,  technologies,  TTP,  etc.  as  the 
baseline  for  experimentation. 

Once  the  objectives  have  been  outlined,  the  team  can 
begin  to  develop  the  specific  questions  to  be  asked  within 
each  of  those  objectives.  These  questions  will  lead  to 
attributes  that  need  to  be  determined  from  the  outcome  of 
the  experiment.  Attributes  will  dictate  what  measures 
(data)  should  be  collected  during  experimentation.  The 
output  from  this  step  should  provide  the  team  with  the 
required  conditions,  events,  and  data  that  must  be 
collected  to  produce  the  required  measures. 

1 .  An  Ideal  Experiment 

Although  running  an  ideal  experiment  is  impossible, 
many  factors  can  be  manipulated  to  make  conditions  as  ideal 
as  possible.  An  ideal  experiment  will  control  all  sources 
of  variation  except  the  variable  of  interest.  In  order  to 
do  this,  the  objectives  will  require  a  theory  that 
identifies  limiting  conditions,  as  well  as  the  casual 
factors  that  influence  the  parameter/quantity  of  interest. 
Additionally,  an  ideal  experimentation  will  measure  all 
active  variables  and  the  limiting  conditions  precisely, 
correctly,  and  reliably. 
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According  to  the  Code  of  Best  Practice  for 
Experimentation,  an  ideal  experiment: 

•  Manipulates  only  one  independent  variable  at  a  time, 

•  Observes  change  in  only  one  dependent  variable, 

•  Excludes,  or  controls  for  all  relevant  extraneous  or 
intervening  variables, 

•  Involves  valid,  reliable,  precise,  and  credible 
measurement  of  all  variables, 

•  Includes  enough  data  collection  opportunities  to 
support  the  inferences  needed  to  translate  findings 
into  actionable  knowledge,  and 

•  Generates  findings,  interpretations,  and  insights. 

In  an  ideal  experiment,  we  are  trying  to  manipulate  a 
single  variable  and  measure  its  impact  on  another  single 
variable  in  order  to  draw  a  cause-effect  relationship. 
Again,  in  real  world  situations  ideal  conditions  are 
sometimes  not  feasible.  Real  world  experimentation 

attempts  to  run  a  larger  quantity  of  experiments  in  order 
to  observe  and  account  for  the  influence  of  outside 
factors.  If  the  experimentation  team  is  not  cautious, 
variables  are  inadvertently  manipulated,  outside  variables 
cannot  be  controlled,  or  the  desired  variable  is  not 
effectively  measured.  These  instances  can  change  an 
experiment's  expected  outcome,  and  possibly  detract  from 
the  general  knowledge  base  on  the  subject  matter. 

Bad  experiments,  which  cloak  weak  or  false 
knowledge  in  an  aura  of  science,  will  make 
mischief,  not  music.  At  a  minimum,  they  will 
slow  the  process  of  understanding  the  impact  of 
innovation  on  transformation  (by  forcing  other 
research  and  experimentation  to  learn  and 
demonstrate  that  they  are  wrong)  and  cost  money 
(by  encouraging  investments  that  will  not  pay  off 
as  expected) .  At  a  maximum,  bad  experiments  well 
lead  to  flawed  mission  capability  packages  that 
fail  in  the  field.  (COBP-Experimentation,  140) 
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C.  CONCLUSION 

Once  the  experimentation  team  has  defined  the 
experiment's  objectives,  they  can  then  begin  to  formulate 
specific  questions.  These  questions  have  matching 
attributes  that  will  provide  linkage  between  the  questions 
within  the  initiative  and  capabilities  in  the  FORCEnet 
Concept  or  tasks  in  joint  documents.  The  experimentation 
team  will  then  determine  what  data  should  be  gathered  to 
validate  these  attributes  and  answer  the  questions. 

Additionally,  each  objective  will  need  to  specify  the 
overall  conditions  under  which  this  data  will  be  gathered. 
In  the  following  step,  the  initiatives  will  be  diagramed 
and  checked  to  ensure  that  all  of  these  components  are 
complete  and  correct. 
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VI .  PROCESS  DIAGRAMS 
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Figure  8.  Process  Diagrams 


Modeling  the  individual  objectives  of  an  experiment 
will  allow  the  team  to  analyze  the  various  relationships 
within  each  focus  area.  Models  highlight  any  inputs, 
variables,  and  processes  when  developing  the  experiment 
ob j  ectives . 

Trident  Warrior  requires  a  high  level  IDEFO  model  be 
developed  for  each  objective  that  shows  how  the  objective 
contributes  to  that  initiative.  For  the  objective  level 
and  below,  more  targeted  models  are  to  be  employed, 
depending  on  the  analytic  intent  of  the  objective. 
Multiple  model  types  can  be  included  within  the  different 
areas  of  research  to  maximize  their  effectiveness.  The 


analysis  team,  modeling  assist  team  and  initiative  leads 
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will  determine  which  models  best  fit  into  each  situation, 
and  specify  what  models  need  to  be  constructed.  Modeling 
is  an  important  step  in  comprehending  the  experiment  and 
its  objectives  and  evaluating  each  objective  for 
thoroughness . 

A.  QUAD  CHARTS 

In  order  to  uniformly  and  effectively  display  the  many 
objectives  within  each  experimentation  campaign,  the 
Trident  Warrior  team  has  developed  its  own  design  diagram, 
called  a  Quad  Chart.  A  Quad  Chart  should  be  organized  for 
each  objective  within  the  experiment,  and  will  "display  all 
pertinent  information  about  an  objective's  purpose  and 
requirements,  provide  an  easy  to  examine  display,  in  a 
format  that  can  be  used  for  presentations,  and  provide 
easily  visualized  information  that  can  be  edited  and 
corrected  before  more  detailed  planning  and  FIRE  input" 
(TW-05  Short  Planning  Directions,  Appendix  A)  .  A  Quad 
Chart  is  useful  because  it  is  a  method  of  describing  an 
entire  objective  in  a  single  diagram. 

A  header  should  be  above  the  Quad  Chart,  containing  a 
one  sentence  initiative  statement  and  the  point  of  contact 
for  that  initiative.  The  first  quadrant  should  define  the 
objective  with  a  single,  over-arching  statement  of  the 
objective's  goal  of  what  is  to  be  learned  or  accomplished. 
The  second  quadrant  should  define  the  recommendations  that 
will  be  presented  to  the  Military  Utility  Assessment  (MUA) 
Board  if  the  objective  is  successful.  Together,  the 
objective  statement  and  the  MUA  recommendation  describe 
what  is  to  be  accomplished  within  this  objective.  The 
third  quadrant  should  contain  the  questions  or  proposals 
that  would  support  the  objective  and  MUA  recommendation. 
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Multiple  questions  may  be  required  per  objective.  The 
collective  set  of  answers  to  these  questions  will  provide 
the  information  to  be  produced  for  this  objective  and  its 
MUA  board. 

Quad 
Page-1 


One  sentence  Initiative  statement  Point  of  Contact 

Objective 

Single  overarching  statement. 

MUA  Recomendation 
Precieved  benefit  if  objective  is 
successful. 

Questions/Proposal 

Multiple  questions,  if  needed,  each 
supporting  an  Information  Goal. 

Operational  Conditions 

Statement  of  all  required  conditions. 

Same  one  sentence  Initiative  statement 

Point  of  Contact 

System  Conditions 

Information  Conditions 

Statement  of  all  required  conditions. 

Statement  of  all  required  conditions. 

Attributes 

Data  Sources 

List  of  all  Measures  needed  to 

List  of  all  types  of  Data  Sources 

answer  Questions,  including  details. 

needed  to  determine  Measures. 

Figure  9.  Quad  Chart 

Quadrants  4  through  6  will  gather  general  conditions 
to  frame  the  situations  and  collect  the  correct 
information.  The  fourth  quadrant  will  contain  a  statement 
of  all  the  required  operational  conditions,  specifying 
which  people  or  organizations  will  be  responsible  for 
conducting  the  operations  within  the  objective.  The  fifth 
quadrant  will  contain  a  statement  of  all  the  required 
system  conditions  for  this  objective.  The  sixth  quadrant 
will  contain  a  statement  of  all  the  required  information 
conditions  for  this  objective,  covering  information  type, 
load,  protocols,  distribution,  etc.  The  seventh  quadrant 
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will  provide  a  general  description  of  the  attributes 
required  to  answer  the  questions  or  meet  the  proposals  in 


quadrant  3.  Finally,  quadrant  8  will  list  all  the  data 
sources  needed  to  determine  the  measures  in  quadrant  7 . 
This  will  include  whether  the  data  is  subjective  or 
quantitative,  as  well  as  whether  this  data  will  be  recorded 
electronically,  through  chat  logs,  in  observation  logs,  or 
through  surveys . 


B .  IDEF  MODELS 

IDEF  or  Integrated  DEFinition  language  is  the 
modeling  technique  used  to  model  functional 
processes  and  corresponding  information  to 
support  Functional  Process  Improvement.  The 
Department  of  Defense  selected  the  IDEF  [model] 
as  the  technique  for  functional  managers  to  use 
for  function  (activity)  and  information  (data) 
modeling.  IDEFO  was  selected  as  the  standard 
technique  for  activity  modeling.  (McConnelly) 

IDEFO  is  used  to  model  the  decisions,  actions,  and 
activities  of  an  organization  or  system,  in  order  to 
communicate  the  functional  perspective  of  a  system.  IDEFO 
models  describe  the  functions  that  are  performed  as  well  as 
what  is  needed  to  perform  these  functions.  The  basic 
structure  used  in  IDEFO  modeling  is  the  referred  to  as  the 
ICOM  format.  ICOM  stands  for  Input,  Control,  Output,  and 
Mechanism.  The  center  square  of  the  ICOM  represents  the 
activity,  or  the  function  to  be  accomplished.  Inputs  are 
all  the  things  that  will  be  transformed  by  the  activity, 
including  materials  and  information.  Controls  are  elements 
that  govern  or  constrain  the  activity,  including  budgets 
and  policy.  Output  is  the  result  of  the  activity. 
Mechanisms  are  the  elements  that  take  part  in  or  support 
the  activity,  to  include  people,  systems,  and  facilities. 
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Constraint 


Figure  10.  Generic  ICOM  for  an  IDEFO  model 
Constructing  a  Context  Diagram  is  the  most  basic 
element  of  the  IDEFO  function  modeling  method  and  will 
represent  the  function  at  its  highest  level,  limiting  the 

scope  of  the  model.  A  functional  decomposition  of  the 

Context  Diagram  will  produce  the  major  activities  required 
to  complete  the  activity  described  in  the  Context  Diagram. 
IDEFO  modeling  is  an  iterative  process  and  should  be 
continued  until  reaching  the  level  of  detail  that  is 
desired.  Subsequent  models  can  continue  to  be  broken  down, 
with  the  new  models  in  each  level  explaining  actions 

required  to  complete  the  actions  of  the  previous  level. 
This  process  will  continue  until  the  IDEFO  model  has  been 
decomposed  into  its  simplest  state.  Based  upon  this 

decomposition,  it  will  be  possible  to  determine  if  any 
inputs,  controls,  outputs,  mechanisms,  or  relationships 
within  the  experiment  were  overlooked.  Then  it  may  be 

necessary  to  update  the  context  diagram  to  include  these 
changes . 
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Figure  11.  A  Context  Diagram  for  Bandwidth  Management 

Objective  (from:  TW  '05) 


C.  OPERATIONAL  SEQUENCE  DIAGRAMS 

The  functional  decomposition  of  IDEF  models  can 
produce  a  complex  output  with  multiple  layers  of 
information.  These  layers  of  information  can  sometimes  be 
confusing  and  difficult  to  follow.  Operational  Sequence 
Diagrams  (OSD)  provide  a  single  page,  flow  chart  view  of 
relationships  between  organizations,  people,  and/or 
technologies  involved  in  a  task.  An  OSD  is  essentially  a 
flow  chart  that  has  been  applied  to  a  functional  process  to 
model  the  sequence  of  data  or  information  as  it  flows 
through  the  system.  An  OSD  can  then  be  used  as  a  planning 
tool  or  an  analysis  tool  when  developing  an  experiment. 
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Figure  12.  OSD  Model  for  Bandwidth  Management  Objective 

(From:  TW  ' 05) 


D .  CONCLUSION 

Without  these  diagrams,  it  is  possible  to  overlook 
relationships  that  may  factor  into  the  scenario.  By 
drawing  process  diagrams  for  each  objective,  the 
experimentation  team  hopes  to  ensure  that  each  objective  is 
complete  and  correct.  This  includes  the  questions 
involving  the  desired  attributes  of  each  objective,  the 
data  needed  to  answer  these  questions,  and  the  overall 
conditions  that  this  data  will  need  to  be  collected  under. 
The  following  steps  will  designate  specific  scenarios  to 
set  the  conditions  for  each  objective  and  set  up  a 
comprehensive  schedule  for  actually  running  the  experiment. 
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VII .  EXPERIMENT  DESIGN/EVENT  DEFINITION 


Phase 

1 

2 

3 

4 
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6 

Due  Dates 

Pre-CDC 

CDC 
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Harvest 

Asset 

Identification 
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Figure  13.  Experiment  Design/Event  Definition 

A.  EXPERIMENT  DESIGN 


The  product  of  experimental  objectives  development 
provides  the  team  with  the  overarching  focus  area,  the 
major  initiatives  within  the  focus  area,  and  the  objectives 
that  support  those  initiatives.  The  objectives  will  now  be 
aligned  with  specific  questions  that  identify  the 
attributes  to  be  gathered  by  the  experiment.  An  attribute 
is  a  general  characteristic  that  can  apply  to  any  number  of 
tasks,  systems  or  processes.  This  enables  attributes  to 
provide  the  link  between  the  objectives.  An  attribute  is 
associated  with  a  certain  capability,  which  is  expressed  as 
a  single  word  (i.e.  timely,  flexible,  redundant)  .  This 
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characteristic  can  be  desired  or  undesired. 


In  order  to 


determine  the  attribute  for  an  objective,  specific  measures 
must  be  taken.  These  measures  comprise  the  data  that  is 
collected.  Thus,  data  is  collected  to  determine  the 
measures  and  answer  the  questions,  which  describe  the 
objectives  that  comprise  the  initiatives,  making  up  the 
overarching  concept  of  the  experimentation  campaign. 


Anatomy  of  an  Experiment 


Figure  14.  Breakdown  of  an  Experiment 
For  example,  if  one  focus  area  of  an  experiment  is 
defined  as  mission  planning,  a  possible  experimental 
objective  could  be  to  develop  the  tools  to  support 
planning.  The  experimentation  team  may  prepare  a  number  of 

questions  or  goals  that  need  to  be  achieved  in  order  to 
satisfy  this  objective.  A  list  of  possible  goals/questions 
for  this  example  could  be:  (1)  Provide  rapid  reach-back, 
(2)  Provide  Complete  Information,  (3)  Provide  a  consistent 
Common  Operating  Picture  (COP),  and  (4)  Provide  easy 
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collaboration.  When  these  goals/questions  are  stated,  it 
becomes  easy  to  determine  the  attributes  required  to  meet 
each  goal  and  satisfy  the  objective.  The  attributes  for 
this  example  would  be:  (1)  rapid,  (2)  complete,  (3) 

consistent,  and  (4)  easy.  The  first  three  objectives  are 
not  difficult  to  comprehend,  and  only  the  fourth  goal  would 
require  further  explanation.  The  measures  required  to 
validate  these  attributes  can  now  be  determined.  It  is 
easy  to  become  distracted  with  the  naming  of  attributes, 
while  the  real  focus  should  be  on  the  measures.  In  order 
to  validate  the  first  attribute  (rapid) ,  it  might  be 
necessary  to  measure  the  quantity  of  time  required  to 
receive  information  from  a  system.  This  measurement  should 
be  defined  as  the  interval  of  time  between  submitting  a 
request  for  information  and  the  receipt  of  that 
information.  Measurements  can  also  be  subjective,  but  it 
is  usually  more  convenient  to  put  this  subjective 
observation  on  a  quantitative  scale  (i.e.  on  a  scale  of  1- 
5,  how  comfortable  were  you  with  the  subject  matter,  with  1 
being  not  comfortable,  and  5  being  very  comfortable) .  This 
method  allows  the  experiment  to  collect  subjective  data, 
and  the  analysis  team  to  perform  a  quantitative  analysis. 
This  subjective  measurement,  or  the  measurement  of  the  time 
delay  in  the  previous  example,  is  the  actual  data 
discovered  by  the  experiment. 

Once  the  details  of  the  objectives  have  been 
established,  the  experimentation  team  can  now  begin  to 
combine  these  objectives  into  a  single  experiment  with 
unique  scenarios  that  will  produce  the  desired  operational 
conditions,  systems  conditions,  and  information  conditions. 
The  Code  of  Best  Practices  for  C2  Assessment  defines  a 
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scenario  as  "A  description  of  the  area,  the  environment, 
means,  objectives,  and  events  related  to  a  conflict  or  a 
crisis  during  a  specified  time  frame  suited  for 
satisfactory  study  objectives  and  the  problem  analysis 
directives."  A  scenario  should  represent  a  plausible  real 
world  situation  that  would  reflect  the  factors  that  are 
needed  for  each  objective.  Scenarios  may  also  require 
smaller  vignettes  in  order  to  produce  all  of  the  required 
conditions  for  each  objective.  "A  vignette  is  primarily 
used  for  smaller  scenarios,  particularly  as  excursions  from 
the  main  scenario"  (C0BP-C2  Assessment,  167)  .  The 

scenarios  and  vignettes  set  the  conditions  and  restrictions 
for  each  objective.  These  conditions  allow  comprehensive 
analysis  and  create  a  structure  where  the  results  of  the 
analysis  can  be  understood  and  interpreted.  "The  purpose 
of  scenarios  is  to  ensure  that  the  analysis  is  informed  by 
the  appropriate  range  of  opportunities  to  observe  the 
relevant  variables  and  their  relationships"  (COBP- 
Experimentation,  200) .  Scenarios  must  be  selected,  crafted 
and  adapted  to  ensure  that  they  support  the  objectives  of 
the  experiment. 

"The  use  of  a  single  scenario,  even  one  that  has  been 
carefully  crafted  to  focus  on  the  key  issues  of  an 
experiment,  invites  suboptimization  and  narrows  the  range 
of  applicability  for  the  findings"  (COBP-Experimentation, 
221)  . 

B .  EVENT  DEFINITION 

1.  Measures  of  Effectiveness 

In  its  simplest  form,  military  experimentation  tries 
to  evaluate  operational  capabilities.  In  order  to  perform 
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these  evaluations,  previous  experimentation  teams  have  used 
descriptions  of  Measures  of  Effectiveness  (MOEs)  and 
Measures  of  Performance  (MOPs) .  The  Defense  Acquisition 
University  defines  a  Measure  of  Effectiveness  (MOE)  as  "a 
measure  of  operational  success  that  must  be  closely  related 
to  the  objective  of  the  mission  or  operation  being 
evaluated, "  and  Measures  of  Performance  as  "measures  of  a 
system' s  technical  performance  expressed  as  a  distinctly 
quantifiable  performance  feature  (i.e.  speed,  payload, 
range,  time,  or  frequency) 

When  attempting  to  relate  MOEs  and  MOPs  to  specific 
items  within  the  experimentation  process,  these  definitions 
can  be  confusing.  MOEs  are  always  a  higher  level  of 
measurement  than  MOPs,  but  exactly  what  they  are  describing 
depends  on  the  context.  Within  our  previous  example  of 
developing  the  focus  area  of  mission  planning  with  an 
experimental  objective  of  developing  tools  to  support 
planning,  there  are  multiple  ways  of  naming  MOEs  and  MOPs. 
The  initiative  of  mission  planning  could  provide  a  Measure 
of  Effectiveness  that  would  describe  the  overall  quality  of 
the  mission  plan.  Under  this  MOE,  a  possible  MOP  would  be 
provided  by  the  objective  in  a  measurement  of  the  quality 
of  the  planning  support  tool  set.  However,  other  potential 
MOEs  are  available.  The  attributes  within  each  goal  could 
provide  a  Measure  of  Performance.  Within  this  example 
those  measures  would  be  rapidity,  completeness, 
consistency,  and  ease.  Similar  to  the  attributes  these 
measures  were  developed  from,  the  fourth  MOP  would  be  the 
most  difficult  to  comprehend. 

Another  option  for  the  Measure  of  Effectiveness  (MOE) 
would  be  the  experimental  objective.  Listed  in  the  last 
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example  as  a  MOP,  this  measure  could  describe  the  quality 
of  the  planning  support  tool  set.  The  MOPs  under  this  MOE 
would  be  the  measures  of  rapidity,  completeness, 
consistency,  and  ease,  as  described  by  the  attributes. 
Regardless  of  which  option  is  chosen,  the  nomenclature  must 
remain  constant  through  the  experiment.  If  MOEs  are 
determined  at  the  initiative  level,  it  must  remain  that  way 
across  the  experiment.  MOPs  abide  by  the  same  restriction, 
if  MOPs  are  described  by  the  attributes,  it  should  be 
constant  across  the  experiment. 

The  actual  level  within  the  experiment  that  the 
MOPs/MOEs  define  is  context  dependant.  The  Modular  Command 
and  Control  Evaluation  Structure  (MCES)  mandates  that  there 
be  rigid  boundaries  between  MOEs  and  MOPs,  but  it  does  not 
say  where  to  establish  those  boundaries. 


Environment 


DP:  Dimensional  parameter 
MOP:  Measure  of  C2 
system  performance 

MOE:  Measure  of  02 
system  effectiveness 

MOFE:  Measure  of  force 
effectiveness 


Figure  15.  Relationships  Among  Classes  of  MOMs  (from:  MCES) 
The  terms  MOE  and  MOP  were  developed  to  provide 
insight  on  the  experiment,  but  they  may  actually  cause  more 
confusion  than  insight.  It  is  possible  to  describe  the 
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experiment  using  only  initiatives,  objectives,  attributes 
and  measures. 

2.  Master  Sequence  of  Events  List 

Once  a  list  of  desired  objectives  with  corresponding 
scenarios  is  produced,  along  with  a  list  of  desired 
measures  to  be  collected,  the  experimentation  team  will 
begin  to  layout  a  schedule  of  events  for  the  actual 
experiment.  This  schedule  of  events  is  called  a  Master 
Sequence  of  Events  List  (MESL)  .  The  MESL  will  combine  the 
experimentation  campaign' s  multiple  scenarios  together  with 
the  schedules  for  the  participating  assets,  to  produce  a 
document  that  delineates  a  time  and  flow  for  all  of  the 
major  activities  within  the  actual  conducting  of  the 
experiment . 

The  MESL  must  first  consider  the  schedule  of  the 

assets  that  will  be  participating  in  the  experiment. 
Within  Naval  experimentation,  this  schedule  will  include 
time  in  port,  time  spent  in  transit,  and  time  on  station. 
Additionally,  this  schedule  will  include  time  for  specific 
events  such  as  flight  operations  and  crew  watch-standing 
rotations.  The  experimentation  team  will  have  limited 
control  over  these  aspects  of  the  schedule. 

Next,  the  MESL  should  consider  the  various  scenarios 

required  to  complete  the  experiment.  Occasionally, 

multiple  scenarios  can  be  run  at  the  same  time,  but  great 
caution  should  be  taken  to  ensure  that  they  do  not 

conflict.  The  various  scenarios  should  be  scheduled  in  an 
orderly  fashion,  and  should  not  impose  undue  levels  of 
stress  upon  the  ship  and  its  crew.  The  conditions  required 
to  answer  the  each  question  need  to  be  present  at  least 
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once  within  the  running  of  the  experiment.  Ideally  there 
would  be  multiple  opportunities  to  complete  each  objective. 
Scenarios  that  correspond  to  higher  priority  objectives 
should  be  run  earlier  in  the  schedule,  permitting  them  to 
be  bumped  down  in  the  case  a  conflict.  If  higher  priority 
scenarios  are  not  run  when  scheduled  and  no  excess  time  is 
available,  a  decision  will  have  to  be  made  on  which  lower 
priority  objectives/scenarios  can  be  omitted.  The 
objective  of  the  MSEL  is  to  answer  all  of  the  questions  for 
each  objective  within  the  given  timeframe. 

C.  CONCLUSION 

Once  the  overall  plan  for  the  experimentation  campaign 
has  been  laid  out,  the  team  will  begin  to  grasp  the  volume 
of  the  task  they  are  undertaking.  It  is  important  that 
every  scenario  has  been  laid  out  in  specific  detail. 
Concentrating  on  one  detail  at  a  time  and  not  becoming 
overwhelmed  by  the  sheer  volume  will  see  the  experiment  to 
completion.  Once  the  scenarios  are  defined  and  developed 
in  detail,  the  experimentation  team  can  begin  to 
concentrate  on  how  and  where  the  individual  pieces  of  data 
will  be  collected. 
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VIII . 


DATA  COLLECTION  PLAN 
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Figure  16.  Data  Collection  Plan 


A  veteran  experimenter  will  require  only  a  quick 
glance  at  the  Trident  Warrior  Experimentation  Process  to 
realize  that  no  specific  step  is  included  to  analyze  data. 
This  is  a  very  dangerous  omission,  as  it  does  not  express 
the  significance  of  the  Data  Analysis  Plan.  Within  Trident 
Warrior,  the  Data  Analysis  Plan  (DAP)  is  written  in 
conjunction  with  the  Data  Collection  Plan  (DCP) ,  creating  a 
Data  Collection  and  Analysis  Plan  (DCAP)  .  This  is  done 
immediately  following  the  Mid-Planning  Conference  (MPC) , 
approximately  4  months  before  the  actual  experiment. 
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A.  DATA  ANALYSIS  PLAN 

What  makes  Trident  Warrior  different  from  other 
naval  assessment  events  is  the  level  of  detail  of 
the  analysis  [planning] .  That  level  of  detail 
can  be  attributed  to  the  Trident  Warrior  Process. 

The  strict  compliance  to  this  process  is  what 
ensures  event  consistency  and  allows  us  to 
maintain  a  high  standard  in  our  FORCEnet 
assessments.  CAPT  Chris  Abbott,  director  of 
FORCEnet  Innovation  and  Experimentation. 
(Poeltler  and  Gallup,  29) 

While  it  may  seem  obvious  that  data  must  be  collected 
before  it  can  be  analyzed,  it  is  not  so  clear  that  you  must 
know  how  the  data  will  be  analyzed  before  you  can  know  what 
data  to  collect.  By  preparing  the  data  analysis  plan 
before,  or  in  conjunction  with,  the  data  collection  plan, 
the  experimentation  team  ensures  that  only  relevant,  useful 
data  is  collected.  "In  economic  terms,  the  data  analysis 
plan  acts  as  the  source  of  demand.  The  data  collection 
plan  acts  as  the  source  of  supply"  (NATO  COBP  for 
Experimentation,  224)  .  Without  the  data  analysis  plan,  it 
becomes  very  tempting  to  simply  collect  the  data  that  is 
easiest  to  access,  rather  than  the  data  that  is  most 
influential  to  the  experiment.  In  order  to  provide  the 
necessary  data,  the  data  analysis  plan  must  be  linked  to 
the  data  collection  plan,  the  post-experiment  modeling 
plan,  and  the  material  necessary  to  revisit,  revise,  and 
perhaps  extend  the  conceptual  model  underlying  the 
experiment.  This  ensures  that  the  data  collected  will  be 
directly  applicable  to  the  overall  purpose  of  the 
experiment . 

The  first  step  of  data  analysis  is  "to  identify  the 
independent  variables  and  their  measures,  then  the  active 
variables  and  their  measures,  and  finally  the  intervening 
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variables  (including  those  that  are  to  be  controlled  in  the 
experiment)  and  their  measures"  (NATO  COBP  for 
Experimentation,  226)  .  A  quick  glance  will  also  show  that 
much  of  the  analysis  is  very  in-depth,  and  it  is  often  a 
requirement  to  have  a  statistical  expert  within  the 
experimentation  team.  Analysis  can  be  categorized  into 
three  general  phases:  descriptive  analysis  of  individual 
variables,  bivariate  analysis  of  relationships,  and 
multivariate  analyses  of  larger  patterns. 

1 .  Individual  Variables 

Once  the  data  has  been  converted  into  the  form 
required  for  analysis  and  formulated  into  an  integrated 
data  set,  the  first  analytic  effort  should  be  a  descriptive 
analysis  of  each  variable  of  interest.  "These  descriptive 
analyses  are  performed  to  (a)  identify  and  correct  data 
anomalies,  (b)  understand  the  distribution  of  each 
variable,  and  (c)  identify  any  transformations  of  those 
variables  with  distributions  that  may  make  analysis 
misleading"  (NATO  COBP  for  Experimentation,  227) .  When 
identifying  and  correcting  data  anomalies,  analysts  should 
search  for  invalid  or  illogical  data.  The  Data  Analysis 
Plan  should  include  time  to  research  these  anomalies  in  the 
original  records  and  correct  them.  Uncorrectable  data  will 
need  to  be  excluded  from  the  data  set.  Next  analysts 
should  review  the  distribution  of  each  variable,  searching 
for  variables  that  would  cause  an  error  within  the 
analysis.  Typical  examples  include  variables  that  have  a 
single  value  or  variables  that  in  only  a  small  number  of 
cases  differ  from  a  single  value.  These  variables  should 
also  be  removed  because  they  will  not  be  able  to  contribute 
to  the  differences  in  the  variables  of  interest. 


Additionally,  in  this  step  analysts  should  search  the  data 
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for  outliers,  or  small  groups  of  data  that  are  very  distant 
from  the  bulk  of  the  data.  Outliers  will  make  the 
application  of  simple  linear  statistics  invalid  unless  they 
can  be  transformed  (possibly  using  a  log  function) , 
excluded  using  the  creation  of  a  new  variable,  or 
subdivided  from  the  bulk  data  using  a  dummy  variable.  The 
last  technique  actually  suggests  that  the  data  is  from  a 
bimodal  distribution. 

2 .  Bivariate  Analyses  of  Relationships 

Bivariate  relationships  are  typically  considered  to  be 
little  more  than  a  building  block  for  multivariate 
relationships.  However,  because  most  hypotheses  are 

started  in  bivariate  terms  (IF  A,  THEN  B,  under  CONDITION 
C)  ,  bivariate  relationships  comprise  a  majority  of  testable 
propositions.  When  attempting  to  identify  relationships 
between  various  factors  and  variables,  the  order  for 
conducting  bivariate  analyses  is  (NATO  COBP  for 
Experimentation) : 

1.  Dependent  variables; 

2.  Control  factors 

3.  Control  factors  and  dependent  variables 

4.  Independent  variables 

5.  Control  factors  and  independent  variables 

6.  Independent  variables  and  dependent  variables 

Dependent  variables  are  tested  first  in  order  to 
ensure  that  they  are  not  correlated.  If  two  dependent 
variables  are  strongly  correlated,  only  one  of  them  should 
be  fully  analyzed.  This  also  means  that  the  underlying 
conceptual  model  should  be  revisited  because  it  incorrectly 
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separately,  as  they  could  prove  important  in  building  a 
comprehensive  model. 

If  a  strong  relationship  between  control  factors  is 
found,  one  of  them  should  be  removed  to  save  efforts  in 
analyzing  similar  relationships. 

By  examining  the  relationships  between  control  factors 
and  dependent  variables,  analysts  can  quantify  the 
effectiveness  of  the  experiment.  In  a  well-designed 
experiment,  control  factors  have  no  direct  impact  on  the 
data  from  the  experiment.  A  major  problem  in  most  human 
subject  experiments  is  learning  through  repetition,  and  an 
experiment  should  be  designed  to  limit  this. 

Next  the  analyst  should  check  for  correlation  between 
the  individual  independent  variables.  Most  analytic  tools 
assume  that  the  independent  variables  are  unrelated,  and 
thus,  having  correlated  independent  variables  will  yield 
false  statistical  explanations.  Correlated  independent 
variables  will  be  handled  in  the  same  manner  as  previous 
variables:  they  can  be  combined,  they  can  be  analyzed 

separately,  or  one  of  them  can  be  thrown  out. 

Similar  to  the  relationship  between  control  factors 
and  dependent  variable,  the  relationship  between  control 
factors  and  independent  variables  could  indicate  a  flaw  in 
the  experimental  design  or  in  the  conceptual  model.  A 
correlation  between  control  factors  and  dependent  variables 
does  not  necessarily  contaminate  the  data,  but  this 
correlation  should  be  considered  when  conducting  the 
multivariate  analyses. 

Finally,  the  purpose  of  the  experiment  is  to  examine 
the  relationship  of  the  independent  and  the  dependent 
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variables.  These  tests  of  correlation  provide  direct 
answers  to  the  hypothesis  posed  in  a  conceptual  model. 

3 .  Multivariate  Analyses 

The  multivariate  analyses  are  the  end  effort  to 
examine  the  data  in  the  full  affects  of  the  surrounding 
environment.  After  running  all  of  the  individual  bivariate 
analyses,  the  experimentation  team  has  ensured  that  the 
effects  on  the  dependent  variables  come  as  a  direct  result 
of  a  change  in  the  manipulated  variable.  Ideally,  this 
will  allow  the  experimenters  to  draw  a  cause-effect 
relationship  between  the  two  variables. 

B.  DATA  COLLECTION  PLAN 

"The  data  collection  plan  includes  all  the  variables 
to  be  collected,  all  the  places  where  they  are  collected, 
all  the  means  of  collection,  and  all  the  places  the  data 
will  be  stored  for  processing"  (COBP  for  Experimentation, 
241)  .  The  data  collection  plan  is  required  to  be 

simultaneously  very  broad  and  also  very  specific.  It  must 
not  only  specify  who,  when  where,  and  how  all  data  is  to  be 
collected,  but  it  must  also  cover  all  training  and  support 
required,  proficiency  standards  and  quality  control.  A 
plan  for  the  archiving  and  processing  of  raw  data  into  a 

form  that  can  be  processed  by  the  data  analysis  plan  must 
also  be  included.  The  key  steps  of  a  data  collection  plan 
include  (COBP  for  Experimentation,  242)  : 

•  Specifying  the  variables  to  be  collected, 

•  Identifying  the  collection  mechanism  for  each 
variable, 

•  Ensuring  access  for  collecting  each  variable, 

•  Specifying  the  number  of  observations  needed  for 
each  variable  and  checking  to  ensure  they  are 
expected  to  be  generated. 
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•  Identifying  the  training  required  to  ensure 
quality  data  collection, 

•  Specifying  the  mechanisms  to  ensure  data  capture 
and  archiving,  and 

•  Defining  the  processes  needed  for  data  reduction 
and  assembly. 

In  addition  to  this  sequence,  there  are  also  multiple 
ways  of  collecting  the  data:  automated  collection, 

recording  for  later  reduction,  surveys,  subject  testing, 
and  human  observation. 

1 .  Automated  Collection 

As  experiments,  particularly  in  the  Command  and 
Control  functions,  become  increasingly  automated,  it 
becomes  more  convenient  to  automate  the  data  collection  as 
well.  Typical  automated  collection  focuses  on  system 

loads;  workstation  loads;  system,  workstation,  and 
application  usage;  as  well  as  a  comparison  between  test 
subject's  perceptions  and  the  "ground  truth."  The  data 
collection  plan  must  specify  where  and  when  this  data  will 
be  collected,  how  the  data  can  be  collected  without 
affecting  the  experimentation  scenario,  how  to  synchronize 
the  data  collection  and  the  experiment,  and  where  the  data 
will  be  archived. 

2 .  Recording  for  Later  Reduction 

Due  to  the  limited  supply  and  abilities  of  human 
observers,  experimentation  teams  often  rely  on  audio  or 
video  recordings  to  supplement  their  data  collection 
process.  Unfortunately,  even  recordings  are  not  always  a 
thorough  means  of  capturing  all  of  the  events  within  an 
experiment.  There  is  the  possibility  of  catching  one  side 
of  a  conversation,  the  test  subject  stepping  out  of  camera 
or  microphone  range,  and  any  number  of  technical  problems 
with  the  recording  equipment.  Unless  the  experimenters  are 
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properly  prepared  for  the  recording  process,  the  material 
they  record  will  likely  be  incomprehensible,  inadequate,  or 
incomplete . 

3 .  Surveys 

Surveys  are  popular  data  collection  tools  because  they 
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familiarity  with  the  equipment,  and  their  familiarity  with 
other  subjects.  Subjects  can  also  be  surveyed  during  the 
experiment  to  determine  their  perceptions,  knowledge, 
ideas,  understanding,  attitudes,  insights,  and  situational 
awareness  as  they  are  being  tested.  The  experimentation 
team  can  also  be  surveyed  during  the  experiment  to  gather 
their  insight  on  how  well  the  systems  are  performing  and 
how  they  can  be  improved. 

Surveys  can  pose  problems  within  an  experiment.  A 
survey  can  lock  the  experimentation  team  into  only 
observing  the  predetermined  items  that  were  mentioned  in 
the  survey.  To  combat  this  problem,  it  is  good  to  include 
some  open-ended  questions  in  your  survey  that  ask  the 
subject  for  additional  reactions  and  observations.  Another 
potential  problem  is  that  surveys  required  time.  Subjects 
will  require  additional  time  to  complete  surveys,  and  this 
time  must  be  scheduled  into  the  master  sequence  of  events 
list.  The  only  way  to  determine  how  much  time  is  necessary 
to  complete  the  survey  would  be  to  pretest  the  survey  using 
similar  respondents.  Using  the  experimentation  team  to 
pre-test  the  survey  will  not  provide  adequate  time  for  the 
less  familiar  test  subjects  to  complete  the  survey. 
Precise  attention  must  be  paid  to  the  wording  of  surveys. 


60 


Misunderstood  or  confusing  questions  will  not  provide  the 
data  desired.  If  a  survey  is  conducted  within  multiple 
phases  of  an  experiment,  it  will  tend  to  impact  future 
behaviors.  In  order  to  prevent  the  data  from  reflecting 
this  "learning  curve"  by  the  subjects,  it  is  sometimes 
beneficial  to  show  the  subjects  the  survey  before  the  first 
phase  of  the  experiment.  The  experimentation  team  should 
also  realize  that  a  poorly  structured  survey  creates 
unnecessary  work  on  the  data  analysis  team.  Multiple 
choice  or  scaled  questions  are  easier  to  process 
analytically.  Surveys  can  be  a  very  effective  means  of 
gathering  data  that  would  be  otherwise  unobservable,  but 
due  to  the  many  risks,  any  experimentation  team  using 
surveys  should  include  a  team  member  with  expertise  in 
survey  design  and  procedures. 

4.  Subject  Testing 

Subject  tests  are  generally  used  to  gather  information 
on  proficiency  or  personal  characteristics.  "Proficiency 
tests  are  used  when  the  results  of  the  experiment  are 
expected  to  depend,  at  least  in  part,  on  the  particular 
skills  of  the  individuals  or  teams  participating.  If 
proficiency  is  a  control  or  is  being  used  to  assign 
subjects  in  order  to  control  it  out  of  the  experiment, 
testing  will  be  needed"  (COBP  for  Experimentation,  249)  . 
In  this  situation,  it  is  beneficial  to  not  release  the 
scores  of  the  proficiency  test  to  the  subjects  until  after 
the  experiment. 

Subject  tests  are  also  used  to  test  for  personal 
characteristics.  Typical  characteristics  tests,  such  as  IQ 
or  personality  tests,  should  be  chosen  from  tests  that  have 
already  been  published  and  validated.  There  is  no  reason 
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for  the  experimentation  team  to  attempt  to  reinvent  these 
tests  and  possibly  give  the  experiment  suspect  credibility. 
These  tests  should  be  administered  professionally.  Data 
from  these  tests  should  not  be  linked  with  the  individual 
subjects  through  processing. 

5 .  Human  Observation 

a.  Observer  Selection 

Observers  are  a  key  part  of  the  data  collection 
process  and  should  have  an  appropriate  background  with 
respect  to  the  experiment's  focus.  Observers  should  have 
some  proficiency  at  observation  and  additional  training 
should  be  available.  Within  Trident  Warrior,  observers  are 
typically  underway  on  vessels,  and  it  is  important  that 
they  blend  in  with  their  environment  and  do  not  cause  an 
unnecessary  distraction.  Observers  must  be  proficient  in 
knowing  what  type  of  data  they  need  to  collect  and  how  to 
collect  it.  It  is  not  uncommon  for  an  observer's  shift  to 
exceed  8  hours,  and  therefore,  observers  should  be 
physically  fit  and  capable  of  handling  the  stress  of  their 
j  ob . 

b.  Observer  Scheduling 

Observer  scheduling  and  positioning  is  a  key 
issue.  Time  should  be  scheduled  for  additional  training 
and  familiarization.  It  is  good  to  incorporate  additional 
time  into  the  schedule  for  observers  to  review  their  notes 
and  format  the  data  for  analysis.  It  is  also  a  good  idea 
to  have  extra  observers  available.  When  they  are  not 
observing,  they  can  be  used  in  a  quality  control  function. 
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C.  CONCLUSION 

Typically,  the  DCAP  should  be  published  well  before 
the  beginning  of  the  experiment,  in  order  to  provide  the 
experimentation  team  and  observers  time  to  familiarize 
themselves  with  the  plan.  However,  within  Trident  Warrior, 
access  to  FIRE  permits  everyone  to  view  the  individual 
elements  that  will  make  up  the  DCAP,  allowing  the  team  to 
shorten  the  timeline  and  make  changes  to  the  DCAP  right  up 
until  the  last  minute. 
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Figure  17 .  Exec 

A.  BEFORE  THE  EXPERIMENT 


1 .  Data  Collection 

Whenever  possible,  the  experimentation  team  should 
collect  any  available,  pertinent  data  before  the  experiment 
begins.  This  is  typically  limited  to  background  and 
expertise  of  the  test  subjects.  Determining  differences 
between  subjects  can  ease  data  analysis  and  lend 
credibility  to  the  experiment.  It  is  important  to  archive 
this  data  immediately,  and  not  become  distracted  by  the 
upcoming  experiment.  If  the  data  is  not  archived  before 
the  experiment  starts,  there  is  a  good  chance  that  it  will 
become  lost  in  the  shuffle.  Data  should  always  be  saved  in 
both  raw  and  processed  forms.  Raw  data  will  permit  the 
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analysis  team  to  review  the  data,  recreate  the  scenario, 
and  even  fix  corrupted  or  incorrectly  processed  data. 
Whenever  dealing  with  personal  data,  it  is  important  to 
ensure  that  all  personal  information  will  be  kept  private. 

Privacy  protection  is  an  important  element  that 
cannot  be  overlooked.  It  is  imperative  that  the 
subjects  understand  that  they  are  not  being 
evaluated  (the  systems,  doctrines,  organizational 
innovations,  under  study  are  being  assessed,  not 
the  subjects)  and  that  no  links  will  be  made  from 
individual  participants  to  scores  and/or  results. 

Not  only  must  the  understand  this,  but  they  must 
also  be  confident  that  this  is  true.  Failure  to 
instill  this  confidence  can  result  in  distorted 
results.  (COBP  for  Experimentation,  259) 

2 .  Training 

a .  Observers 

Most  data  cannot  be  collected  using  an  automated 
collection  program,  and,  therefore,  will  require  the  use  of 
observers  to  be  recorded.  Observers  must  understand  why 
they  are  collecting  data,  and  what  purpose  that  data  will 
serve.  Explaining  any  underlying  methodologies  and 
theories  will  empower  the  observer  and  significantly 
increase  their  understanding  and  data  collection  abilities. 

A  training  program  should  be  created  that  focuses 
on  when,  how  and  where  the  data  is  to  be  collected. 
Observers  should  be  familiarized  with  the  processes 
affiliated  with  observation,  surveys,  questionnaires,  and 
instrumentation,  in  accordance  with  the  Data  Collection  and 
Analysis  Plan.  Observers  should  be  given  live  scenarios  in 
which  they  can  practice.  A  proficiency  test,  comprised  of 
a  written  exam  and  a  practical  exam,  should  be 
administered.  The  written  exam  should  test  the  observer's 
knowledge  of  the  fundamentals,  while  the  practical  exam 
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should  present  the  observer  with  a  scenario  that  will  test 
their  ability  to  correctly  collect  and  code  data.  If 
necessary,  a  scheduled  review  session  should  be  conducted. 

Jb.  Subjects 

Military  experimentation  typically  compares  the 
current  Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel  or  Facilities  (DOTMLPF)  with  a 
proposed  alternate  solution.  The  subjects  will  be  more 
familiar  with  the  current  DOTMLPF  than  the  proposed 
alternative,  giving  it  an  unfair  advantage.  New  systems 
require  the  greatest  quantity  of  "familiarity  training." 
Subjects  need  to  know  how  they  interface  with  the  new 
system,  what  tasks  it  performs,  how  it  performs  them,  how 
the  system  uses  the  underlying  data,  what  operations  the 
system  performs  on  the  data,  how  the  data  is  displayed,  and 
the  sources  of  the  data  (COBP  for  Experimentation) . 
Subjects  will  need  to  understand  their  roles  and  duties 
within  the  experiment.  As  done  with  the  observers,  a 
proficiency  test  should  be  given  to  test  the  subject's 
familiarity  with  the  proposed  DOTMLPF  solution.  Initial 
training  of  subjects  and  observers  should  proceed 
independently,  but  a  final  segment  of  training  should  be 
run  with  the  observers  gathering  data  on  the  subjects,  as 
the  subjects  run  through  a  scenario. 

B.  CONSIDERATIONS  FOR  OBSERVERS 

Observers  can  be  utilized  as  either  silent  observers 
or  to  administer  surveys  and  questionnaires.  Regardless, 
it  is  imperative  that  they  do  not  influence  the  outcome  of 
the  experiment  by  engaging  the  subjects  in  conversation, 
expressing  opinions,  or  interfering  with  the  experiment  in 
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any  other  way.  They  must  not  provide  hints  or  suggestions, 
even  when  something  has  been  overlooked. 

Incidents  will  occur  where  failure  to  provide 
certain  information  could  cause  the  experiment  to 
go  drastically  awry.  To  account  for  those 
instances,  procedures  should  be  established  for 
the  observers/controllers  to  get  advice  from  the 
senior  member  of  the  observation  team  who,  in 
turn,  should  confer  with  the  experiment  director 

and  senior  controller.  (COBP _ for 

Experimentation) 

Only  experiment  controllers,  and  not  observers,  should 
answer  questions  or  offer  any  item  of  substance  concerning 
the  experiment. 

An  observer's  shift  should  never  exceed  8  consecutive 
hours.  A  fatigued  observer  will  not  be  able  to  collect 
pertinent  data  with  the  speed  and  accuracy  of  a  well-rested 
observer.  It  is  important  that  different  observers  code 
data  in  the  same  manner,  so  as  to  prevent  confusion  when 
the  data  is  processed.  Observer  shifts  should  be  well 
planned,  and  should  ensure  that  no  data  is  lost  during 
handoff.  Observer  shifts  should  occur  on  different 
schedules  from  the  subject's  shift  changes.  This  will 
allow  the  observer  to  note  the  process  and  exchanges 
between  subjects. 

The  Data  Collection  Plan  should  describe  how 
comprehensive  the  observer  must  be  and  should  specify  which 
areas  need  to  be  covered  in  great  detail.  Otherwise,  the 
observer  will  become  overwhelmed,  attempting  to  collect 
more  data  than  is  manageable. 

Data  collectors  and  observers  should  be  constantly 
supervised.  Their  collected  data  should  be  checked  for 
completeness  and  confidence  in  its  validity.  Observers 
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should  be  discreetly  alerted,  without  notifying  the 
subjects,  of  events  likely  to  affect  their  station.  This 
will  often  result  in  the  need  for  regularly  scheduled 
meetings  and  timely  interaction. 


C. 


DATA  COLLECTION  AND  PROBLEM  SOLVING 


Automated  collection  systems  and  instrumentation 
can  fail,  humans  bring  their  own  backgrounds  and 
foibles  to  the  program  and,  regardless  of 


training,  some  will  try  to 
is  interesting,  rather 
assigned  to  collect.  To 
the  experiment,  it  is 
collectors  be  supervised 
mechanism  is  in  place 
necessary  data  is  being 
Experimentation,  268) 


collect  what  they  feel 
than  what  they  are 
ensure  the  success  of 
imperative  that  data 
and  a  quality  control 
to  ensure  that  the 
collected.  (COBP  for 


1 .  Automated  Data 

Automated  data  can  be  taken  from  screen  captures, 
email  archiving,  requests  for  information,  and  database 
snapshots  by  using  a  simple  computer  program  to  dissect 
this  data  and  immediately  archive  it.  Unfortunately, 
because  it  is  so  easy  to  use,  problems  with  automated  data 
collection  often  go  unnoticed.  This  data  must  be  validated 
at  regular  intervals  to  ensure  that  the  expected  data  is 
being  processed  in  the  expected  manner.  The  validation 
interval  should  vary  with  respect  to  the  data  flow  rate. 
Additionally,  the  ease  with  which  this  data  can  be 
collected  could  lead  the  collection  team  to  gather  large 
amounts  of  unimportant,  unexpected  data,  overwhelming  the 
data  analysis  team. 

2.  Loss  of  Personnel 

One  of  the  most  difficult  obstacles  for  an 

experimentation  team  to  overcome  is  when  a  key  individual 

is  unable  to  participate  in  the  experiment.  Most  commonly, 
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this  lost  participant  was  scheduled  as  either  a  test 
subject  or  as  an  observer.  Whenever  possible,  it  is  a  good 
idea  to  run  extra  iterations  of  the  experiment,  so  that  a 
fault  in  one  run  will  not  bring  the  entire  experiment  to  a 
halt.  It  is  also  a  good  idea  to  train  extra  participants 
so  that  test  subjects  are  replaceable.  Additionally,  it 
may  be  wise  to  schedule  extra  time  to  train  these 
replacement  subjects  on  the  DOTMLPF  solution  being 
analyzed.  Finally,  if  no  other  options  are  available,  the 
experiment  may  have  to  continue  shorthanded.  The  loss  of 
manpower  may  affect  performance,  and  this  should  be 
accounted  for  in  the  data  analysis. 

In  the  case  of  an  observer  dropping  out  of  the 
experiment,  there  are  a  few  additional  options.  It  would 
be  convenient  if  there  were  extra  observers  on  hand  that 
could  step  in  and  fill  the  position.  It  might  also  be 
possible  for  a  supervisor  to  take  on  the  additional  role  of 
observer.  It  may  be  necessary  to  shift  some  observers  from 
less  important  areas  to  fill  the  void. 

3.  System  Failures  and  Data  Contamination 

The  best  solution  to  avoiding  system  failures  is  to 
design  redundancy  into  the  experiment  and  avoid  single 
points  of  failure  within  the  systems.  All  equipment  should 
be  tested  prior  to  the  beginning  of  the  experimentation 
campaign.  A  common  error  with  most  system  testing  is  not 
subjecting  the  systems  to  the  same  loads  they  will  be 
forced  to  endure  during  the  experiment,  resulting  in  system 
failures.  It  is  important  to  test  the  systems  with  the 
identical  loads  and  traffic  they  will  encounter  during  the 
experiment . 
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As  with  any  experiment,  the  observation  of  test 
subjects  will  inherently  cause  them  to  work  harder  and 
strive  to  excel.  Ideally  this  factor  would  be  mitigated  by 
the  observation  of  teams  on  both  the  system  (or  idea)  being 
tested,  as  well  as  the  system  (or  idea)  that  is  currently 
in  use.  Data  may  become  contaminated  due  to  members  of  the 
experimentation  team  influencing  the  outputs  or  even  test 
subjects  intentionally  skewing  the  data.  In  any  case,  if 
data  becomes  contaminated,  it  is  necessary  to  exclude  this 
data  from  the  analysis. 

4 .  Premature  Reporting 

It  is  often  very  tempting  to  attempt  to  draw 
conclusions  during  the  experimentation  phase.  Data 
analysis  must  always  follow  data  collection  and  precede  any 
type  of  reporting.  The  experimentation  team  cannot  get  off 
schedule  and  attempt  to  provide  instant  feedback  on  the 
results.  "Instant  feedback  can  be  misleading  and  lead 
people  to  believe  that  they  do  not  need  to  execute  the  data 
analysis  plan"  (COBP  for  Experimentation,  275) .  An  attempt 
to  provide  immediate  reporting  tends  to  prevent  good 
analysis  and  consistent  results.  It  is  very  difficult  to 
refocus  a  decision-maker' s  attention  away  from  the 
immediate  report,  even  when  the  data  analysis  proves  the 
immediate  report  to  be  false. 

D .  DATA  ARCHIVING 

Before  the  data  can  be  analyzed,  it  should  be  archived 
in  its  unprocessed  form.  Raw  data  will  enable  the 
experiment  to  be  replicated  and  validated,  thus  permitting 
it  to  contribute  to  knowledge  generation.  The  importance 
of  preserving  the  raw  data  cannot  be  overstated,  and  it 
should  be  saved  in  more  than  one  location  and  on  more  than 
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one  medium.  Additionally,  all  steps  involved  with 

processing  and  manipulating  the  data  should  be  recorded, 
allowing  the  final  analysis  to  be  completely  reconstructed. 

The  handoff  of  the  data  between  data 
collectors/observers  and  the  data  processors/analysts  can 
be  a  major  point  of  failure  within  the  experiment.  Both 
the  data  collectors  and  the  analysts  should  be  involved  in 
the  conversion  of  raw  data  into  the  form  required  by  the 
analysts . 

Data  [processing]  should  occur  as  soon  after  the 
data  collection  as  practical,  preferably  in  the 
days  immediately  following  the  experiment.  This 
has  the  benefit  of  performing  the  data 
[processing]  while  experiment  events  are  still 
fresh  in  the  minds  of  the  data  collectors.  It 
also  has  the  added  benefit  of  having  all  the 
collectors  together  so  that  they  can  work  as  a 
team.  (COBP  for  Experimentation,  273) 
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X.  REPORTING  RESULTS /CONCLUSION 
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Figure  1 8 . 

REPORTING  RESULTS 


Reporting  Results 


1 .  Final  Report 

The  Experimentation  team  should  attempt  to  complete 
its  final  report  on  the  experiment  approximately  three 
months  after  execution.  The  Final  Report  will  be  a 
complete  documentation  of  the  experiment. 

a.  Principle  Results 

The  Final  Report  should  consist  of  at  least  4 
major  sections  along  with  an  executive  summary.  The  first 
section  should  outline  the  principle  results  of  the 
experiment.  The  principle  results  section  should  then  be 
broken  down  into  the  different  initiatives  of  the 
experiment,  and  each  individual  initiative  comprising  a 
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small  section  of  the  report.  The  layout  should  feature  a 
short  description  on  the  conditions  of  experiment,  followed 
by  a  short  description  of  the  quantity  that  was  measured. 
This  should  be  followed  by  the  conclusions  drawn  from  the 
measurement  and  straightforward  recommendations  that  are 
suggested  due  to  the  outcome  of  this  initiative.  These 
initiatives  should  be  grouped  together  with  similar 
initiatives  that  fall  under  the  same  objective. 

Jb .  Ba  ckgro  un  d 

Following  the  principle  results  section,  the 
Final  Report  should  now  provide  a  brief  history  of  the 
experiment.  This  should  include  a  short  history,  an 
overview  of  activities  within  the  experiment,  a  brief 
description  of  the  experiment's  overall  concept,  and  an 
overview  of  the  operations  that  took  place  within  the 
experiment . 

c.  Initiative's  Context 

The  initiative's  context  section  should  contain 
an  overview  of  the  conditions  of  the  experiment  quality, 
the  context  of  the  overarching  focus  area,  and  context 
descriptions  of  the  experiment  initiatives. 

d.  Experimentation  Status  and  Recommendations 

The  last  section  of  the  Final  Report  should 

provide  a  general  idea  on  the  outcome  of  the 
experimentation  campaign  as  a  whole.  It  should  include  the 
general  status  of  the  experiment,  as  well  as 
recommendations  from  each  significant  observation  within 
the  experiment. 

Also  included  in  the  Final  Report  should  be  an 
executive  summary  that  provides  a  quick  overview  of  the 
information  contained  within  the  rest  of  the  report. 
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2. 


Other  Reports 

a.  Operational  Agent  Assessment 

The  Operational  Agent  Assessment  (OAA)  is  a 
report  that  is  unique  to  the  Trident  Warrior  program.  The 
culminating  step  of  the  experimentation  process  is  a 
Military  Utility  Assessment  Board,  where  a  committee  of 
high-level  officers  makes  recommendations  based  upon  the 
results  of  the  experiment.  In  order  to  avoid  subjecting 
the  MUA  board  to  a  comprehensive  Final  Report,  the  Trident 
Warrior  team  drafts  an  Operational  Agent  Assessment.  The 
Operational  Agent  Assessment  is  tailor-made  especially  for 
the  MUA  board,  and  streamlines  the  Final  Report  into  a  few 
comprehendible  points  on  each  major  focus  area  within  the 
campaign.  The  purpose  of  this  report  is  to  inform  the 
high-level  decision  making  of  the  MUA  board.  The  OAA  is 
drafted  by  the  experimentation  team,  and  then  presented, 
along  with  the  Final  Report,  to  the  MUA  board  by  the 
operational  agent  sponsoring  the  experiment  (NETWARCOM  for 
TW)  . 

Jb.  Military  Utility  Assessment 

Once  the  Final  Report  is  complete,  a  Military  Utility 
Assessment  Board  is  convened  to  determine  how  the  results 
and  conclusions  of  the  experiment  will  benefit  the  military 
and  ultimately  the  warfighter.  The  Military  Utility 

Assessment  Board  will  publish  the  Military  Utility 
Assessment  that  lists  recommendations  for  changes  to 
current  Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel,  and  Facilities  policy. 

Within  the  Navy,  the  completed  MUAs  are  then  briefed 
to  the  Sea  Trial  Executive  Steering  Group,  who  will  rate 
each  recommendation  with  one  of  four  possible  outcomes. 
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The  first  possible  outcome  would  be  to  classify  the 
experiment  results  as  providing  a  significant  enhanced 
warfighting  capability  impact.  It  would  then  be  presented 
to  the  Commander  of  U.S.  Fleet  Forces  Command  (CFFC)  for 
immediate  delivery,  insertion  into  the  Naval  Capability 
Development  Process  (NCDP) ,  or  passed  on  to  Navy  Warfare 
Development  Command  (NWDC) .  A  second  possible  outcome  for 
a  recommendation  from  the  MUA  would  find  the  experiment 
results  provide  a  positive  enhanced  warfighting  capability 


impact . 

In 

this 

case. 

the  recommendation  would 

be 

forwarded 

to 

CFFC, 

and  its 

progress  would 

be  carefully 

tracked . 

A 

third 

outcome 

would  be  to 

classify 

the 

experiment  results  as  not  mature  and  requiring  further 
examination.  The  event  would  then  be  rescheduled  for 
additional  experimentation  as  warranted.  The  final  option 
would  be  to  classify  the  recommendation  as  not  providing  an 
impact,  and  the  event  would  then  be  closed  out. 

B.  CONCLUSION 

The  overall  purpose  of  military  experimentation  is  to 
provide  a  competitive  advantage  to  the  warfighter.  By 
testing  new  technologies  and  innovative  methods  of 
employing  them,  the  experimentation  team  hopes  to  prove  or 
disprove  the  added  benefit  of  their  use.  An  experiment  is 
not  deemed  a  success  even  if  the  MUA  board  does  not 
recommend  the  solutions  being  tested.  Success  is  dependent 
upon  running  a  comprehensive  and  effective  experiment  that 
highlights  the  various  advantages  and  disadvantages  of 
differing  military  doctrine,  organization,  training, 
materiel,  leadership,  personnel,  or  facilities  (DOTMLPF) . 
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The  Trident  Warrior  process  has  successfully  seen 
multiple  experimentation  campaigns  to  completion.  By  using 
a  proven,  reliable  experimentation  process,  they  have 
multiplied  the  experimentation  campaign' s  chances  of 
success,  and  also  increased  the  reliability  and  confidence 
of  others.  "I  have  come  to  rely  on  Trident  Warrior 
information  and  assessments"  VADM  James  McArthur,  commander 
NETWARCOM.  Enabling  decision  makers  to  make  informed, 
intelligent  decisions  on  the  future  of  the  military  is 
really  what  it  is  all  about. 
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APPENDIX  A.  TW-05  SHORT  PLANNING  DIRECTIONS 


Conceptually,  military  operational  experiment  planning 
is  simple.  An  objective  is  defined  and  that  leads  to 
measures,  then  to  data  collection  needed  to  provide  those 
measures.  The  experiment  results  provide  a  determination 
of  the  level  off  success  with  the  objective.  But,  the 
devil  is  in  the  details.  Careful  planning  is  needed  to 
ensure  that  results  address  objectives.  If  you  wish  to 
compare  the  capabilities  of  two  systems,  or  processes,  and 
test  one  under  one  set  of  conditions  and  the  other  under 
different  conditions,  useful  comparisons  will  not  emerge. 
Successful  planning  proceeds  along  well-defined  steps,  each 
producing  an  essential  planning  element.  This  document 
explains,  briefly  and  simply,  these  experiment  planning 
elements . 

Planning  an  experiment  is  carried  out  in  two  phases: 

Initiative  planning  is  concept  planning.  It  sets  the 
overarching  structure  for  the  experiment,  defines  the 
goals,  what  is  to  be  learned  and  what  quantities  are  needed 
to  do  it. 

Experiment  planning  provides  experiment  details:  how 
it  is  to  be  carried  out,  what  systems  are  to  be  used  and 
where,  specific  data  to  be  captured,  where,  and  how  that  is 
to  be  done . 


Initiative  Planning  Structure 

An  Initiative  is  described  by  a  single  sentence  that 
applies  to  everything  that  is  to  be  done  under  its 
umbrella.  Each  Initiative  has  a  set  of  one  or  more 
Objectives,  with  each  Objective  containing  the  following 
components : 
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Objective 
Information  Goal 

This  pair  of  elements  describes  what  is  to  be  accomplished 

Questions 

They  are  the  bridge  between  Information  Goals  and  Measures  and  Data 

Conditions  (3) 

Specific  conditions  may  be  needed  to  match  and  address  the  Objective. 
They  and  Data  to  be  captured  lead  to  the  MSEL. 

Measures 

Data 

Quantities  that  must  be  determined  to  answer  the  Questions. 

Types  of  Data  needed  to  produce  the  Measures 

The  following  will  present  planning  structures  and 
element  descriptions,  first  with  broad  descriptions  then 
increasing  detail  as  the  document  proceeds.  This 

presentation  order  is  followed  so  that  overarching 
understanding  can  be  developed  before  a  lot  of  detailed 
reading  is  done.  The  order  of  presentation  is: 

•  Quad  Charts  -  Initiative  Elements 

•  Initiative  Planning  Components 

•  F.I.R.E.  Initiative  Input 

•  Threads  and  Numbering 

•  Quad  Charts  to  F.I.R.E.  Initiative  Input  Mapping 

•  Experiment  Planning  and  Data  Capture  Plan 

•  Experiment  Planning  Components 

•  FORCEnet  Attributes  and  Measures 

•  Planning  Stoplights 

•  Amplifying  Comments  on  Initiative  Planning  Components 

•  Amplifying  comments  on  Experiment  Planning  Components 


Quad  Charts  -  Initiative  Elements 

There  is  a  single  set  of  two  Quad  Charts  for  each 
Objective.  These  Charts  serve  the  following  purposes: 

•  Display  all  pertinent  information  about  an  Objective's 
purpose  and  requirements. 

•  Provide  an  easy  to  examine  display,  in  a  format  that 
can  be  used  for  presentations . 

•  Provide  easily  visualized  information  that  can  be 
edited  and  corrected  before  more  detailed  planning  and 
F.I.R.E  input. 

There  is  a  Quad  Chart  review  process  where  they  are 
scrutinized  carefully  to  ensure  congruence  across  the 
planning  elements  and  their  accuracy  and  sufficiency. 
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A  Quad  Chart's  elements  are  (content  descriptions  are 
later  in  this  document) : 


Header  -  Contains  Initiative  statement  and  POC. 

The  colors  here 
are  chosen  to 
match  those  in 
the  following 

Objective  -  1st  quadrant  -  One  Quad  set  per  Objective 

Info  Goal  -  2nd  quadrant  -  This  is  the  Objective’s  pair. 

Questions  -  3rd  quadrant  -  Multiple  Questions  allowed. 

Questions  address  the  Info  Goals. 

FIRE  input 
diagram. 

Conditions  -  4th,  5lh,  and  6th'  General  rather  than  specific  descriptions. 

Yellow  is  used 

Measures  -  7lh  quadrant  -  General  descriptions  of  Measures  types. 

Units  and  other  specifics  not  necessarv. 

for  three  elements 
because  of  their 
close 

relationships. 

Data  -  8th  quadrant  -  General  description  of  types  of  Data  capture. 

Error ! 

There  may  be  multiple  Questions,  Measures,  and  Data 
and  the  associations  between  them  is  not  necessarily  clear 
in  the  Quad  Charts.  When  information  is  input  to  F.I.R.E., 
care  must  be  taken  to  provide  these  associations.  This  is 
described  in  the  later  section  on  Quad  Charts  to  F.I.R.E. 
input  mapping. 


F.I.R.E.  Initiative  Input 

Placing  the  Initiative  information  in  the  F.I.R.E. 
knowledge  management  system  serves  the  following  purposes: 

•  Segment  Initiative  information  input  into  experiment 
Threads  that  will  be  used  for  detailed  planning. 

•  Provide  additional  detail  over  that  in  the  Quad  Charts 
to  enable  Experiment  Planning. 

•  Archive  information  in  a  structure  that  provides  easy 
association  between  elements  for  analysis  and  results 
production . 

•  Provide  a  structure  for  semi-automated  results 

reporting . 

The  main  differences  between  F.I.R.E  information  and 
the  Quad  Charts  are: 

•  F.I.R.E.  segments  Questions  into  individual  Threads. 

•  Conditions,  Measures,  and  Data  descriptions  are  more 
detailed  in  F.I.R.E. 

•  Conditions  may  be  more  specific  in  F.I.R.E.  to  relate 
to  their  specific  Question. 
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•  Multiple  Measures  and  Data  specifications  are  sorted 
into  those  that  apply  to  a  specific  Question. 


F.I.R.E.  Initiative  Input 


Initiative 

1“ 

Objective- 1 


Information  Goal 

V  1 

f 

Question- 1 

0-2 

i 


Obj-2 


One  sentence  statement  of  Initiative’s  Purpose  during  this 
experiment.  There  will  be  several  Initiatives. 

One  sentence  statement  of  specific  Objective  Goal  that 
supports  the  Initiative’s  Purpose. 

•  An  hiitiative  will  contain  several  Objectives. 

•  Goals  may  relate  to  operational  capabilities. 

Types  of  Information  needed  to  address  the  Objective  Goal. 

•  Only  one  Information  Goal  statement  per  Objective. 

•  More  than  one  type  of  information  may  be  required. 

•  Includes  guidelines  to  Conditions. 


Questions  provide  links  between  Objectives/Information  and  Measures. 

•  Several  may  be  needed  to  link  to  all  required  Measures. 

•  The  collective  set  of  answers  will  provide  the  Information  to  be 
produced  for  that  Objective. 

•  A  Question  statement  must  lead  directly  to  Measures. 


Cond 


i  t 


Conditions:  situations  to  must  be  set  up  to  obtain  the  correct  Information. 

•  Operational  -  people/organizations  conducting  the  operation 

•  System  -  which  ones  and  their  state 

•  Information  -  what,  load,  protocols,  distribution,  etc. 


Measures 

A,  B,  C... 

Meas 

D,  E... 

.  1 

Data  |  Data  | 

: _  i 

Inputs  for 

Experiment  Planning 


Measures  are  specific  quantities  to  be  produced  to  answer  the  Question. 

•  Included  are  appropriate  units. 

•  Whether  they  are  subjective  or  quantitative  is  stated. 

Types  of  Data  to  be  obtained,  not  details  of  where,  when,  and  how. 

•  Stated  whether  electronic,  human  information  logger,  survey,  Chat 
log,  etc.,  are  to  be  obtained. 

•  State  whether  subjective  or  quantitative. 

•  Operational  units  at  which  Data  is  to  be  gathered. 


Threads  and  Numbering 

Planning  Thread:  a  complete  set  of  definitions. 

Initiative  through  Question.  There  can  be  only  one 

Question  per  Thread,  and  the  question  is  the  ultimate 
definer  of  the  Thread. 


Numbering 
Initiative  Code 

Objective/Information  Goal  number 
Question  number 


Network  Initiative  Example 

NET 

NET.2  2nd  Objective  in  Initiative 
NET.2.3  2nd  Objective,  3rd  Question 


There  is  only  one  Initiative  statement.  Each  of  the 
Objectives  within  an  Initiative  also  has  a  single  statement 
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that  is  repeated  for  each  of  its  Questions.  A  Thread's 
Question  must  be  a  single  question;  multiple  questions  are 
not  allowed  because  proper  measures  can't  be  defined. 

Quad  Charts  to  F.I.R.E.  Initiative  Input  Mapping 

The  diagram  on  the  following  page  shows  how 
information  is  transferred  from  the  Quad  Charts  to  the 
F.I.R.E.  Initiative  input.  It  is  provided  as  a  visual, 
quick  reference  for  F.I.R.E.  inputs.  Descriptions  of  these 
planning  components  are  in  following  sections.  This 
section  only  shows  the  mapping  from  one  to  the  other. 

As  noted  above,  there  are  two  Quad  Charts  per 
Objective.  In  the  figure,  lines  with  balls  at  the  end  show 
that  the  Objective  and  Information  Goal  statements  are  a 
pair . 


Same:  indicates  that  the  information  in  the  Quads  and 
FIRE  are  identical.  This  is  the  case  for  Objective  and 
Information  Goal. 

Expand:  the  information  may  be  exactly  the  same,  or  is 
essentially  the  same  but  may  need  some  expansion  to  be 
specific  for  that  Thread.  This  applies  to  the  three 
Conditions . 

Sort:  the  information  in  that  quadrant  has  to  be 

sorted  to  apply  to  its  individual  Thread.  This  applies  to 
sorting  Measures  and  Data  so  they  are  associated  correctly 
with  the  Questions. 

•  There  can  be  multiple  Questions  and  each  of  them 
defines  a  Thread. 

•  There  can  be  multiple  Measures  and  they  must  be 
correctly  associated  with  the  Questions  they  will  help 
answer.  A  Measure  can  apply  to  more  than  one 
Question . 

•  There  can  be  multiple  Data  and  they  must  be  correctly 
associated  with  the  Measures  they  will  produce.  More 
than  one  type  of  data  may  be  needed  to  produce  a 
Measure . 

•  The  association  of  Question  to  Measures  and  Data  can 
be  one-to-many  and  association  of  Measures  and  Data 
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can  be  many-to-many .  If  there  are  too  many 
associations,  the  Question  is  too  broad.  If 
Objectives  and  Questions  are  defined  narrowly,  it  is 
possible  that  there  will  be  only  one  Measure  and  type 
of  Data  for  each  Question. 


Quad 

Page-1 


Quad 

Page-2 


One  sentence  Initiative  statement 

Point  of  Contact 

Objective  • - 

- •  Information  Goal 

Single  overarching  statement. 

Single  overarching  statement. 

Multiple  requirements,  if  needed. 

Questions 

Operational  Conditions 

Multiple  questions,  if  needed,  each 
supporting  an  Information  Goal. 

-Q-l  Q-2  Q-3 

|  | 

Statement  of  all  required  conditions. 

same 

same 

Same  one  sentence  Initiative  statement 

Point  of  Contact 

System  Conditions 

Information  Conditions 

Statement  of  all  required  conditions. 

Statement  of  all  required  conditions. 

Measures 

Data 

List  of  all  Measures  needed  to 

List  of  all  types  of  Data  needed  to 

answer  Questions,  including  details. 

determine  Measures. 

Ml  M2  M3  M4 

D 1  D2 

'  | 

D3  D4  D5 

!  ; .  : . 

i  8  a 

““  sort 

1 - ! - L 

sort 

Thread  Num 


Objective 


Info.  Goal 


Q-l 


Oper.  Cond. 


Syst.  Cond. 


Info.  Cond. 


Measures 

Ml 

M2 


Data 
D  1 
D2 


sort 


sort 


Thread  Num 


Objective 


Info.  Goal 


Q-2 


Oper.  Cond. 


Syst.  Cond. 


Info.  Cond. 


Measures 

■H  M3 


Data 
D  1 
D2 


I- -I 


Thread  Num 


Objective 


Info.  Goal 


Q-3 


Oper.  Cond. 


Syst.  Cond. 


Info.  Cond. 


Measures 

M2 

M4 


Data 
D3  D4 
D5 


expand 


FIRE  Input 
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Experiment  Planning  and  Data  Capture  Plan 

Experiment  Planning  involves  working  out  all  of  the 
details  of  Data  capture  and  publishing  a  Data  Capture  Plan 
that  guides  conducting  the  experiment.  The  following 
figure  illustrates  the  components  of  the  Data  Capture  Plan. 


Data  Capture  Plan 


Architectures 

MSEL 

Systems 

Personnel 

Processes 

Data  Capture  Instruments 


Info  Logs 


Surveys 


System 
Data  Logs 


Architectures  include  OSDs  and  IDEFs.  Specification 
of  the  systems  to  be  used  is  done  by  a  Systems  Table  and  an 
Installs  Table  that  tracks  progress  toward  making  the 
systems  available. 

Experiment  Planning  Components 

Experiment  Planning  is  grouped  into  categories,  with 
components  within  each  category.  The  following  shows  the 
categories  and  components  and  the  type  of  input  for  FIRE 
required  for  entering  them.  Replicated  means  that  that 
information  is  provided  automatically  from  the  Initiative 
inputs.  Y/N  refers  to  a  Yes/No  drop-down.  Text  refers  to 
a  text  box  where  narrative  input  is  provided. 
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Thread  Information 

This  is  general  information  about  the  Thread.  Most  of 

Thread  Number 

(Replicated) 

it  is  replicated.  A  drop-down  is  used  to  indicate 

Thread  POC 

(Text) 

whether  the  information  will  be  classified. 

Question 

(Replicated) 

Measures 

(Replicated) 

Classified 

(Y/N) 

It  is  at  this  point  that  the  connection  to  FORCEnet 

Fn  Attrib/Meas 

(Text) 

Attributes  and  Measures  is  made. 

Events 

Events  describe  those  things  that  are  to  occur  when 

Data  Locations 

(Text) 

the  data  is  being  captured.  MSEL  (or  other)  events  to 

Event  Times  /  MSEL  (Text) 

be  used  are  listed.  The  exact  location(s)  for  data 

capture  are  listed. 

Electronic  Data 

(Y/N) 

Electronic  data  is  that  captured  directly  by  a  system 

Providing  System 

(Text) 

being  used  in  the  operation  or  inserted  to  capture  data. 

Data  Available 

(Y/N) 

Feasibility  check. 

Collection  Medium  (Text) 

Needed  is  whether  or  not  the  data  will  be  logged, 

Data  Formats 

(Text) 

what  collection  medium  will  be  used  to  record  it,  the 

Time  Stamps  Avail  (Y/N) 

format  used,  and  whether  synchronized  time  stamps 

Electronic  Data  POC  (Text) 

will  be  recorded  and  who  has  direct  responsibility. 

Chat  Logs 

(Y/N) 

C2  and  task  coordination  information  is  carried  out 

Collection  Medium  (Text) 

via  Chat.  These  logs  often  provide  very  useful  data. 

Chat  POC 

(Text) 

Observation  Logs 

(Y/N) 

Personnel  are  used  to  observe  operations  and  log 

Logger  Station 

(Text) 

events. 

Forms  Completed 

(Y/N) 

Loggers  Assigned 

(Y/N) 

They  are  placed  at  defined  locations  and  use  forms 

Logging  POC 

(Text) 

designed  for  their  particular  data  capture. 

Survey  Data  (Y/N) 

Surveyed  Personnel  (Text) 

Form  Completed  (Y/N) 

Administrators  (Text) 

Survey  Medium  (Text) 

Survey  POC  (Text) 


Subjective  opinions  are  gathered  via  surveys. 

Survey  instruments  that  meet  the  Measures 
requirements  are  designed  and  administered  to  a  focus 
group  by  an  assigned  administrator.  Surveys  are  often 
completed  using  electronic  capture. 


Additional  Comments  (Text)  Any  comments  that  are  needed  to  clarify  or  direct 

any  aspect  of  experiment  conduct. 
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The  normal  situation  in  an  experiment  is  that  there 
will  be  data  capture  that  applies  to  more  than  one  Thread. 
Observers,  instruments,  logs,  etc.,  can  and  should  be 
shared  across  Initiatives  and  Objectives.  These  detailed 
planning  components  will  reflect  this  in  that  the  same 
people  will  appear  as  having  responsibilities  in  several 
Threads . 

FORCEnet  Attributes  and  Measures 

It  is  important  that  experiment  results  be  mapped  to 
official  FORCEnet  Attributes  and  Measures.  It  is  also 
important  the  experiment  measures,  which  will  often  be  more 
detailed  than  the  broader  FORCEnet  measures,  be  developed 
naturally  without  influence  from  higher-level  measures. 
Thus,  correlation  between  the  two  is  not  done  until  the 
Experiment  Planning  phase.  Note  in  the  above  section  that 
there  is  a  place  for  entering  this  information.  If  there 
is  no  appropriate  FORCEnet  Measure,  if  appropriate,  the 
Measuree(s)  developed  should  be  nominated  for  addition  to 
the  FORCEnet  Measures. 

Planning  Stoplights 

Red/yellow/green  stoplights  are  provided  for  each 
Initiative.  They  encompass  both  Initiative  and  Experiment 
planning.  There  is  a  light  for  each  of  the  major  planning 
components : 
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The  example  shows  all  except  Data  of  Initiative 
Planning  green  and  Experiment  planning  red. 

Stoplight  colors  are  controlled  by  both  the  Initiative 
Lead  and  Analysis  Reviewer.  Initiative  colors  are 

determined  by  whether  they  meet  the  factors : 

•  Correctness  of  the  input  (meets  standards  for 

content) 

•  Sufficiency  of  the  input  (to  enable  description  of 
succeeding  components) 

•  Congruence  between  Initiative  elements 


87 


Experiment  planning  components  are  judged  solely  on 
completeness.  The  Reviewer  will  send  reasons  for  judgments 
to  the  Lead  by  e-mail. 


Amplifying  Comments  on  Initiative  Planning  Components 

The  following  additional  comments  about  the  Planning 
Components  are  presented  to  better  define  what  is  reguired. 

Examples  of  Planning  Components  will  be  provided  under 
separate  cover  for  further  clarification  of  the  information 
to  be  provided. 

Ob j  ectives :  Several  types  of  Objective  are 

appropriate : 

•  Develop  or  enhance  an  operational  capability. 

•  Develop  a  system  or  test  a  system's  capabilities. 

•  Develop  or  modify  a  process  (normally  involving  human 
operators)  . 

•  Test  TTP  or  SOP,  or  documents  that  direct  operators. 

All  of  these,  and  others,  are  appropriate. 

The  Objective  statement  must  clearly  state  the  exact 
goal,  what  is  to  be  learned  or  accomplished. 

Information  Goal:  An  Objective  and  its  Information 

Goal  are  a  pair.  There  is  only  one  Information  Goal 
statement  per  Objective. 

•  There  may  be  more  than  one  type  of  information  that  is 
directly  related  to  the  Objective. 

•  The  statement  lists  the  types  of  information  needed, 
not  specifics  about  them.  They  are  guidelines  to  the 
more  specific  Measures  and  Data  to  be  obtained. 

•  The  statement  includes  guidelines  for  the  types  of 
conditions  under  which  the  information  is  to  be 
produced . 

Questions :  A  bridge  is  needed  between  the  somewhat 

general  Information  Goal  statement  and  the  specific 
Measures  to  be  produced. 
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A  question  is  needed  for  each  of  the  information  types 
in  the  Information  Goal. 


•  Each  question  defines  an  experiment  Thread. 

•  Multiple  Questions  are  not  allowed,  those  that  contain 
either  explicitly  or  implied  more  than  one  question. 

•  A  Question  should  be  stated  in  a  way  that  leads 
directly  to  the  Measures  to  be  produced. 

Conditions :  It  is  extremely  important  to  have  the 

proper  conditions  set  up  if  appropriate  results  are  to  be 
produced . 

•  Data  obtained  under  improper  conditions  will  yield 
results  that  don't  answer  the  posed  question,  or 
answer  incompletely,  or  worse  yet  provide  incorrect 
answers . 

•  Conditions  lead  directly  to  the  MSEL. 

•  Specified  Conditions  will  be  needed  for  one  or  all  of: 
Operational,  System,  Information. 

For  example:  If  the  objective  is  to  determine  the 

maximum  throughput  that  can  be  achieved  with  a  particular 
system  or  process,  the  information  load  must  be  large 
enough  so  that  the  system  can  be  stressed.  This  would  be 
an  Information  Condition. 

Measures :  Answering  the  Question  requires  that 

specific  Measures  be  produced.  These  are  the  quantities 
that  will  be  reported  as  experiment  results  in  support  of 
the  answer  to  the  question. 

•  Each  must  be  a  specific  quantity,  with  units  if 
appropriate . 

•  There  can  be  more  than  one  Measure  for  a  Question.  If 
there  are  several,  it  may  be  that  more  than  one 
question  is  needed.  Reporting  will  be  cumbersome  if 
there  are  too  many  Measures  associated  with  a  single 
Question . 
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•  A  Measure  must  lead  directly  to  the  Data  acquisition 

needed  to  determine  it.  This  means  providing  some 

specifics  that  provide  guidance  to  data  capture. 

For  Example:  "Average  processing  time"  is 
insufficient.  "Average 

time  to  process  BDA  requests  by  the  sensor 

manager"  contains  the 
needed  specifics. 

•  If  it  has  not  been  done  sufficiently  in  Information 

Requirement,  specify  the  operational  unit,  even  the 
type  of  personnel,  if  they  are  to  be  involved  in 
producing  supporting  Data. 

•  The  Measure  definition  should  imply  the  type  of 

analysis  to  be  done  to  reduce  the  data. 

When  stating  Measures,  care  must  be  taken  to 
ensure  that  needed 

Data  can  be  captured,  and  analyses  done, 
that  will  allow  them  to 

be  determined.  It  is  not  uncommon  that 

Measures  are  stated  that 

cannot  be  obtained  within  the  realities  of 
an  experiment. 

Data :  These  are  somewhat  general  Data  definitions. 

Specific  details  of  where,  when,  and  how  they  are  captured 
are  provided  in  Experiment  Planning.  In  Initiative 

Planning,  sufficient  definition  is  provided  to  allow 
subsequent  details  to  be  completed. 

•  Data  definitions  specify  which  types  of  Data  are  to  be 
obtained : 

o  system  electronic  data  logs 
o  Chat  logs,  etc. 
o  human  data  loggers 

o  post-  or  during-experiment  surveys 

•  These  definitions  alert  planners  to  the  need  to 
produce  particular  Data  collection  instruments. 

For  example:  the  above  Measures  example 
would  lead  to  "Logs  of 
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It  is  not  necessary 


sensor  requests  at...", 
to  specify  who  collects  the 

logs  or  their  format.  This  is  done  in 
Experiment  Planning. 

•  Correlations  between  Measures  and  Data  do  not  need  to 
be  specified.  They  should  be  obvious  if  the 

specificity  referred  to  above  has  been  provided. 


Amplifying  Comments  on  Experiment  Planning  Components 

Correlations  between  the  Data,  Measures  to  be  produced 
that  answer  the  Questions,  (the  Threads)  are  contained  in 
Initiative  Planning.  The  focus  in  Experiment  Planning  is 

on  obtaining  the  Data. 

Experiment  Planning  specifies  the  physical  things 
that  must 

be  done  during  the  experiment,  in  sufficient 
detail,  that  all 

the  needed  systems,  people,  processes,  and  what 
they  are  to 

do,  can  be  in  place  to  produce  the  required  Data. 

Refer  in  what  follows  to  the  former  section  on 

Experiment  Planning  Components.  The  Bold  words  in  its 
figure  are  those  that  appear  in  the  stoplight  and  provide 

the  subsections  below. 

Thread :  Most  of  the  information  provided  is 

replicated . 

•  Thread  numbers  that  are  entered  in  the  Initiative 

input  will  be  replicated  here  and  an  input  form 

provided  for  each  Thread. 

•  The  POC  information  is  not  replicated  because 
additional  people  may  be  involved  in  detailed 
Experiment  Planning. 

•  The  FORCEnet  Attribute  and  Measures  numbers  are 
entered . 

Events :  The  purpose  of  this  information  is  to  allow 
those  responsible  for  data  capture  to  be  in  place  and  ready 
to  capture  data  at  the  appropriate  times. 


91 


•  There  may  be  several  events  during  the  experiment  that 
will  provide  the  needed  data. 

•  There  may  be  more  than  one  data  capture  location,  and 
different  types  of  data  may  be  captured  at  different 
locations.  Specifics  are  needed  as  to  what  data 
where . 

•  It  may  be  necessary  to  identify  a  particular  watch 
station  within  an  organization. 

•  It  may  be  necessary  to  identify  a  particular  system 
from  which  data  is  to  be  obtained. 

•  The  MSEL  will  be  referred  to  for  most  dates  and  times 
for  events.  It  the  event  is  off-MSEL  this  should  be 
stated  along  with  any  specifics  that  will  help 
identify  when  the  event  will  occur. 

Data :  Four  types  of  data  can  be  captured: 

•  Electronic 

•  Chat 

•  Observation  Logs 

•  Surveys 

The  section  for  each  data  type  is  headed  by  a  Y/N 
dropdown.  This  is  to  indicate  whether  that  type  of  data  is 
required.  If  the  answer  is  no,  there  is  no  need  to  provide 
any  further  input  for  that  data  type. 

The  Experiment  Planning  input  forms  in  F.I.R.E  are  set 
up  so  that  there  is  no  background  color  if  no  input  is 
provided  for  an  element.  Thus,  for  a  data  type  not 
required  that  whole  section  will  be  white  (except  for  the 
Y/N  answer) . 

Electronic  Data:  These  data  are  captured  directly  by 

a  system  being  used  during  the  experiment. 

•  Each  system  that  will  provide  such  data  must  be 
identified.  The  specific  data  within  the  system  to  be 
captured  must  also  be  identified. 

•  It  is  often  the  case  that  no  automated  logs  are  either 
available  or  turned  on  for  this  data  capture.  This 
must  be  determined  and  the  Y/N  is  provided  for  status. 
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•  It  is  important  to  know  the  medium  used  for  data 

capture.  The  medium  must  be  compatible  with 

subsequent  data  reduction  capabilities. 

•  Understanding  data  formats  is  important  for  subsequent 
data  reduction.  The  format  should  be  specified  and 
determination  made  as  to  whether  it  is  compatible  with 
reduction  methods. 

•  It  is  normally  assumed  that  systems  have  time  stamps 

available  for  all  data  output,  but  this  often  the 
case.  This  must  be  checked.  Timing  is  crucial  for 

many  Measures.  It  must  also  be  determined  whether  or 
not  time  synchronization  procedures  have  been 

established . 


Chat  Logs :  Communications  that  are  carried  out  via 

Chat  can  provide  either  direct  or  supporting  data. 

•  It  must  be  insured  Chat  will  be  logged  by  the  system 
being  used,  that  personnel  are  assigned  to  capture 
Chat  logs,  and  that  appropriate  recording  media  will 
be  in  place. 

•  Insure  that  there  is  a  means  to  synchronize  Chat  log 
times  with  other  time  logs  and  that  logs  will  have 
time  stamps. 

Observation  Logs :  Observations  are  logged  by  an 

observer  assigned  to  a  specific  location. 

•  The  specific  station  to  be  monitored  must  be  stated. 

•  Logging  forms  will  be  specific  to  the  information  to 
be  obtained  at  each  location.  Stated  should  be  the 
specific  information  to  be  obtained  and  what  measures 
this  information  will  be  used  to  produce. 

•  Logs  must  contain  the  time  at  which  a  "happening"  is 
observed . 

Survey  Data:  Surveys  are  the  principal  means  for 
obtaining  subjective  information  about  a  process  or  system. 
This  information  is  primarily  opinions. 
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•  Specific  personnel  to  be  surveyed  must  be  described. 

•  Surveys  will  be  specific  to  the  information  to  be 

obtained.  Stated  should  be  the  specific  information 

to  be  obtained  and  what  measures  this  information  will 
be  used  to  produce. 

•  Surveys  can  be  administered  by  paper  or 

electronically.  The  means  for  administering  and  for 
post-completion  archival  should  be  stated. 

Required  Documents :  Most  detailed  planning 

information  appears  in  other  documents,  MSEL,  Systems, 
etc.,  not  in  the  Experiment  Planning  forms.  These  other 
documents  are  used  as  references  during  experiment 
planning.  Conversely,  experiment  planning  also  produces 
these  documents.  They  are  a  crucial  part  of  planning  and 
it  is  not  be  possible  to  complete  planning  until  they  are 
complete.  The  way  the  planning  forms  are  set  up,  it  will 
not  be  possible  to  complete  them  until  these  required 
documents  are  also  completed. 
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Experimentation  Management  Guide 


Purpose :  This  task  list  provides  a  decision  support 
and  management  tool  for  experimentation  sponsors,  executive 
agents  or  action  officers,  and  task  managers.  It  is 
intended  to  support  planning,  preparation,  and  execution  of 
experimentation  within  the  context  of  an  experimentation 
campaign.  It  is  intended  primarily  as  a  guide  to  support 
individuals  and  teams  to  identify  the  analytical  activities 
(the  what's)  needed  to  develop  the  experimentation 
framework,  and  a  set  of  process  mechanics  (the  how's)  to 
achieve  desired  outcomes.  It  is  also  intended  to  assist 
senior  decision  makers  in  addressing  tradeoffs  associated 
with  an  experimentation  campaign  over  extended  time 
horizons,  involving  complex  environments,  multiple 
stakeholders,1  competing  research  interests,  and  limited 
resources . 


Background :  This  task  list  complements  the  Code  of 
Best  Practice  for  Experimentation  ( COBPE )2  developed  by  the 
Command  and  Control  Research  Program  (CCRP)  of  the 
Assistant  Secretary  of  Defense  for  C3I.  The  COBPE  provides 
an  overarching  approach  to  the  conceptualization,  design, 
and  execution  of  individual  experiments  or  experimentation 
campaigns,  and  was  developed  to  investigate  evolving 
operational  concepts  and  other  areas  of  interest.  The  COBPE 
recognizes  that,  as  a  practical  matter,  the  dynamics  of 
experimentation  are  influenced  by  many  factors  beyond  the 


1  A  stakeholder  is  anyone  who  has  specific  interest  in  the  results  of 
the  experimentation  campaign.  A  sponsor  is  a  stakeholder  with 
specifically  defined  roles  in  providing  resources  to  conduct  the 
experimentation  campaign. 

2  Code  of  Best  Practice  for  Experimentation,  David  S.  Alberts, 
Richard  E.  Hayes  (Washington,  D.C.:  CCRP,  2002) . 
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control  of  the  experimenters,  especially  in  large  events 
with  long  planning  and  preparation  lead  times. 

The  work  reported  here  was  based  on  the  Multi-INT 
Experimental  Checklist  developed  by  a  team  led  by  Annette 
Krygiel.  That  effort  had  developed  an  intelligence-related 
checklist  based  on  the  COBPE .  The  present  work  represents 
an  effort  to  extend  the  Multi-INT  Checklist  to  apply  to 
defense-related  experimentation  efforts  in  general. 


Format :  This  task  list  provides  the  elements  of  an 

iterative  cycle  (as  shown  in  Figure  1)  combining 
development  and  assessment  activities  leading  to  the 
conduct  of  experimentation  events.  Critical  to  the  success 
and  relevance  of  each  experiment  or  event  is  for  the 
process  to  adapt  to  changes  in  priorities,  environment,  and 
stakeholder  interests  over  the  course  of  event  preparation. 
There  are  two  elements  to  the  experiment:  planning  and 
execution.  The  planning  process  includes  three  critical 
components : 

1.  Establishing  an  Experimentation  Framework 

2.  Planning  and  Design 

3.  Development  and  Validation 

Execution  also  includes  three  components: 


4.  Preparation  and  Rehearsal 

5 .  Execution 

6.  Analyses,  Evaluation,  and  Transition 


It  is  important  to  note  that  many  of  the  experiment 
development  and  execution  activities  are  cyclical  and 
overlapping  rather  than  sequential.  Figure  1  provides  one 
way  to  visualize  this  process.  The  process  must  be  flexible 
enough  to  revisit  components  as  needed.  It  should  also  be 
noted  that  as  environmental  conditions  change  and 
stakeholder  interests  are  affected,  options  should  be 
established  for  continued  participation  or  the  entry  of  new 
participants.  The  checklist  is  directed  mainly  at 
individual  experiments,  but  the  planning  process  described 
in  Section  1  applies  equally  well  to  the  planning  of  an 
experimentation  campaign. 
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Figure  1.  "The  Iterative  Cycle  for  Experimentation" 

1.  Establishing  an  Experimentation  Framework 

1.1  Articulate  and  establish  research/investigative 
ob j  ectives . 

1.1.1  Generate  an  agreed  upon  and  commonly  understood 
statement  of  problem (s)  to  be  investigated. 

1.1.2  Ensure  empowered  representation  from  the 
constituents  of  the  defined  community. 

1.1.3  Ensure  the  objectives  of  the  investigation  or 
experiment  are  measurable,  and  are  clearly,  simply, 
and  soundly  stated. 

[These  steps  may  be  accomplished  in  a  workshop  or  planning 
conference . ] 

1.2  Analyze  alternatives. 

1.2.1  Determine  and  consider  alternative  investigative 
methods  to  experimentation  in  terms  of  products, 
cost/benefit,  feasibility,  and  risk. 

1.2.2  Determine  viable  approaches  to  investigate  areas 
of  interest,  i.e.,  discrete  events  and  linkages  with 
other  experiments  or  research  activities. 

[Careful  articulation  of  information  needs  can  facilitate 
obtaining  data  from  other  research  activities  or 
confirming  that  a  gap  exists.] 

1.2.3  Determine  requirements  for  repeatability, 
validity,  and  credibility. 

1.2.4  Decide  to  conduct  the  experiment ( s )  including 
identification  of  needed  resources  and  resource 
providers . 
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1.3  Identify  the  community  of  interest. 

1.3.1  Identify  stakeholders  and  potential  sponsors  as 
well  as  their  expectations  and  concerns. 

1.3.2  Identify  "senior  leadership"  for  the  event,  i.e., 
the  decisionmaker  for  abort  criteria. 

[It  is  particularly  important  for  senior  leadership  to  be 

kept  informed  of  the  range  of  expectations  and  agendas 
represented  by  the  various  stakeholders.] 

1.3.3  Define  the  "total"  environment  -  research, 
operational,  political,  technical. 

1.3.4  Determine  range  of  possible  effects  of 
investigation  at  technical,  operational,  programmatic, 
cultural,  and  other  levels  (e.g.,  development  of  new 
knowledge,  changed  organizational  structures, 
programmatic  resources)  . 

1.3.5  Develop  a  program  of  peer  reviews,  involving 
independent  evaluations  by  knowledgeable  personnel  of 
the  technical,  programmatic,  and  operational  merits  of 
particular  aspects  of  the  experiment  or  the  campaign. 

1.3.6  Identify  peer  reviewers  and  enlist  their 
participation . 

1.4  Identify  interests  of  stakeholders. 

1.4.1  Identify  points  of  common  interest  and  points 
where  interests  do  not  converge,  in  order  to  define  a 
deconf liction  process. 

[The  experimentation  campaign  planners  must  accommodate  the 
concerns  of  those  who  will  be  affected  by  the 
experimentation  process  and  its  results.] 

1.4.2  Define  the  level (s)  of  interest  for  process 
results— national ,  local,  service,  joint,  limited 
objective,  campaign. 

1.4.3  Define,  with  senior  leadership,  the  purpose  and 
desired  product  types  for  the  process,  to  avoid 
excessive  focus  on  the  output  of  only  one  event  or 
experiment . 

1.4.4  Establish  strategic,  campaign  level  focus  linking 
multiple  events.  Emphasize  cumulative  development  of  a 
body  of  knowledge  that  does  not  depend  on  the  results 
of  only  one  experiment  or  research  activity. 

1.5  Engineer  flexibility  into  the  process. 

1.5.1  Define  critical  decisions  (rather  than 

milestones)  to  allow  for  confirmation  of  sponsor  and 
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stakeholder  interests,  commitment  to  the  course  of 
action,  and  opportunities  for  the  graceful  exit  of 
stakeholders  who  no  longer  wish  to  be  part  of  the 
event  as  well  as  entry  of  new  stakeholders. 

1.5.2  Determine  constraints  on  the  experiment  or  on  the 

campaign  that  impact  iteration/refinement  cycles  (see 
Figure  1 ) . 

[Reviews  should  be  provided  to  senior  leadership  to  ensure 
that  the  ramifications  of  decisions  and  directions  are 
recognized. ] 

1.6  Scope  the  effort. 

1.6.1  Obtain  general  agreement  on  the  type  of 
experiment ( s )  to  be  conducted  e.g.,  discovery, 
hypothesis  testing,  and  demonstration, . 

[In  principle,  it  may  be  advisable  to  change  the  experiment 
type  during  the  development  process.  This  is  a  major 
decision  requiring  careful  assessment  and  stakeholder 
participation. ] 

1.6.2  Derive  the  simplest  experimentation  approach 

lab  test,  modeling  and  simulation,  limited  field  test, 
etc.  -  to  deliver  desired  results. 

1.6.3  Document  the  responsibilities  of  each  participant 
and  obtain  a  common  understanding  and  agreement  among 
the  participants. 

1.6.4  Identify  and  execute  any  required  contractual 
vehicles . 

1.6.5  Determine,  plan,  and  program  to  ensure 
availability  of  required  prototypes,  systems, 
materiel,  databases,  and  infrastructure. 

1.6.6  State  desired  outcomes,  being  explicit  in  terms 
of  goals,  metrics,  applications,  and  relevance  to  type 
of  experiment. 

1.7  Establish  strategic  framework. 

1.7.1  Identify,  assess  the  relevance  of,  and  pursue 
cooperative  joint  efforts  with  complementary  research 
programs  including  those  in  military  services/ j oint 
environments,  government  agencies,  academe,  research 
and  development  organizations,  and  other  venues. 

1.7.2  Develop  contingency  planning  options  and 
procedures  in  anticipation  of  unexpected  events  and 
disruptions . 

1.7.3  In  the  case  of  incremental  funding,  ensure 
resource  commitment  and  scheduling. 
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1.7.4 


Establish  tracking  and  management  mechanisms, 
e.g.,  funding  expenditures  and  calibrate  against 
projected  costs  for  the  experiment,  development  of 
needed  infrastructure  and  capabilities,  task 
execution . 

1.7.5  Develop  an  experiment  schedule  to  address  all 
phases  and  resource  requirements. 

1.7.6  Assess  and  account  for  potential  impact  of 
experimentation  environment  on  experimentation  process 
and  outcomes. 

1.7.7  Initiate  actions  to  secure  required  funding, 
resources,  and  approvals. 

1.8  Identify  requirements. 

1.8.1  Determine  minimum  resources  (personnel, 
facilities,  time)  to  execute  the  experiment. 

1.8.2  Identify  unique  requirements  for  experiment 
conduct  and  design  (infrastructure)  to  include: 

Data  collection  team  qualifications/assignments. 

Data  collection  team  training. 

Instrument  development. 

Consistency  of  involvement  in  experiment  preparation 
period . 

Selection  of  experimentation  infrastructure 
(live/modeling  and  simulation  supported) . 
Identification  of  all  factions  to  be  represented, 
e.g.,  Blue/Friendly,  Red/Hostile,  White/Neutral, 
Green/Third  Party,  etc. 

1.8.3  Identify  unique  requirements  for  experiment 
participant  (subject)  selection  and  preparation  to 
include  desired  expertise,  training  requirements, 
preparation  timelines,  language  proficiency, 
experience  sets,  and  Service  or  Combatant  Command 
representation . 

1.8.4  Determine  required  approvals  to  conduct 
experiments,  e.g.,  review  board  approvals  for  use  of 
human  participants  in  experimentation. 

1.8.5  Schedule  a  policy  review  early  to  identify 
security  issues,  such  as  need  for  accreditation,  or 
policy  waivers. 

1.8.6  Determine  requirements  for,  and  initiate 
processes  to  obtain,  waivers,  such  as  that  of  security 
policy . 

1.9  Commit  resources. 
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1.9.1 


Gain  commitment  of  research  sponsor  (s)  and  degree 
of  support  through  written  agreement  with 
participating  agencies,  i.e.,  ensure  proper  staff 
availability  and  secure  needed  resources  and/or 
funding . 

1.9.2  Assign  and  ensure  commitment  of  Experiment 
Director/Action  Officer  through  the  research  sponsor. 

1.9.3  Document  requirements  in  experiment  objectives, 
requirements,  and  expected  outcomes  with  research 
sponsor . 

1.9.4  Ensure  early  identification  and  commitment  of  all 
experiment  participants  by  their  respective 
organizations  -  customers,  process-owners,  subject 
matter  experts,  prototype  developers,  facilities- 
owners,  trainers,  security  personnel,  etc. 

1.9.5  Document  facility  requirements  and  obtain 
agreements  for  their  provision  with  hosts  and 
supporting  organizations  to  ensure  scheduling  and 
availability. 

2.  Planning  and  Design 

2.1  Define  role  of  the  particular  experiment  within  the 
campaign 

2.1.1  Assess  current  state  of  knowledge  in  terms  of 
what  the  campaign  has  answered  so  far  and  identify 
remaining  unresolved  issues 

2.1.2  Inform  senior  leadership  and  confirm  research 
priorities . 

2.1.3  Define  and  decide  the  next  step  in 
experimentation  or  investigation. 

2.2  Formulate  experiment. 

2.2.1  Review  results  of  research  relevant  to  the 
intended  experiment,  including  reviews  of  studies, 
experiments,  customer  and  process-owner  input,  etc. 

2.2.2  Identify  the  products  and  implications  of  the 
experiment,  such  as  residual  assets,  evaluation, 
requirements  for  acquisition,  new  business  processes, 
etc . 

2.2.3  Ensure  that  the  experiment  incorporates 
appropriately  mature  elements,  such  as  existing 
prototype  versions,  surrogates  or  mock-ups,  and 
modeling  and  simulation. 

2.2.4  Submit  the  experiment  plan  for  peer  reviews  or 
independent  assessments. 
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2.2.5 


Develop  hypotheses  (IF,  THEN  statements)  that  are 
generated  and  structured  to  contribute  to  knowledge 
about  the  concept,  capability,  system,  or  process 
being  investigated. 

2.2.6  Assess  prototypes,  cost,  risk,  and  schedule 

acceptability  constraints  (for  experimentation) . 

2.3  Plan  the  analysis  and  evaluation  methodology. 

2.3.1  Identify  control  variables  and  determine  means  to 
treat  them  adequately. 

[All  variables  should  be  identified  by  type  (independent, 
dependent  and  control) .  Both  manipulation  and  control 
are  integral  to  hypothesis-testing  experiments.  The 
dependent  variables  are  observed  systematically  under 
specified  conditions  while  the  factors  considered  to 
cause  change  -  the  independent  variables  -  are  varied. 
Other  potentially  relevant  factors  must  be  held 
constant,  either  empirically  or  through  statistical 
manipulation.  (Extracted  from  Code  of  Best  Practices 
for  Experimentation  by  Dr.  David  S.  Alberts.)] 

2.3.2  Determine  whether  a  documented  baseline  for 
comparison  exists  or  the  means  to  generate  one  in  time 
for  the  conduct  of  the  experiment. 

2.3.3  Ensure  that  the  evaluation  method  selected  is 
relevant  to  the  type  of  experiment,  select  appropriate 
metrics,  and  determine  options  to  populate  them 
through  data  collection. 

2.3.4  Ensure  participants  for  experiment  have  desired 
expertise,  training  requirements,  preparation 
timelines,  language  proficiency,  experience  sets,  and 
Service  or  Combatant  Command  representation. 

2.3.5  Ensure  processes  and  organizational  changes  are 
considered  in  the  evaluation. 

2.3.6  Ensure  that  the  plans  for  scenario  generation  are 
sufficiently  timely  and  that  they  span  the  conditions 
of  interest  to  the  sponsors. 

2.3.7  Develop  Analysis  Plan,  framework,  and  reporting 
requirements . 

2.3.8  Develop  data  collection  plan(s)  for  processes 
during  the  experiment  and/or  across  the  range  of 
experiments  within  the  campaign  to  ensure  sufficient 
data  to  support  the  evaluation.  Consider  means  to 
ensure  that: 

Data  collected  reflect  critical  indicators. 

Quality  control  processes  are  included 
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Data  collection  allows  for  re-use  and  archiving 
through  file  collection  and  indexing 
Standard  data  formats  are  used  where  possible, 
including  date/time  references. 

The  evaluation  plan  addresses  how  to  interpret, 
generalize,  or  scale  the  results  of  the  experiment. 

2.3.9  If  modeling  and  simulation  will  be  used  to 

validate  and  expand  experiment  findings,  then  ensure 
that  appropriate  models  and  simulation  capabilities 
are  available,  or  planned. 

2.4  Plan  experiment  and  develop  experimental  architecture. 

2.4.1  Determine  and  ensure  understanding  of  applicable 
technical  standards  that  must  be  developed  and/or 
applied  where  feasible. 

2.4.2  Determine  if  existing  experimentation 
infrastructure  in  other  experimentation  and  research 
venues  such  as  Joint  Forces  Command  ( JFCOM) ,  service 
laboratories,  and  academic  institutions  can  be 
used/ leveraged . 

2.4.3  Identify  and  describe  any  legacy  system 
enhancements  or  needed  inf rastructure/tool 
development . 

2.4.4  Identify,  schedule,  and  ensure  commitment  by 
their  owners  and/or  sponsors  of  required  prototypes, 
systems,  materiel,  databases,  and  infrastructure, 
including  all  Government  Furnished  Equipment  (GFE)  and 
Government  Furnished  Information  (GFI)  by  specific 
dates . 

2.4.5  Ensure  appropriate  means  of  technical  control 
over  infrastructure  development,  such  as  configuration 
management  processes. 

2.4.6  Ensure  engineering  resources  are  available  as 
needed  to  develop  the  end-to-end  architecture  to 
support  the  experiment. 

2.4.7  Map  appropriate  elements  of  operational  concept 
or  problem  being  investigated  to  scenario.  State 
research  questions  and  framework  for  reporting  results 
per  the  analysis  plan. 

2.5  Conduct  facility  planning. 

2.5.1  Determine  and  schedule  facilities  and  resources 

required  for  experiment  in  terms  of  manpower, 
equipment,  infrastructure,  etc.  in  accordance  with  the 
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2.5.2 


needs  of  each  specific  phase  of  the  experiment  and  by 
specific  dates. 

Identify  and  resolve  potential  conflicts  with 
other  events  supported  by  the  experimentation  facility 
such  as  training,  integration  and  test,  rehearsal,  and 
experiment  conduct. 

2.5.3  Assess  facility  layout  and  determine  means  to 

minimize  or  avoid  distractions  and  disruptions,  such 
as  from  visitors  and  observers. 

2.6  Develop  training. 

2.6.1  Determine  training  criteria  and  required 
standards  of  proficiency. 

2.6.2  Plan  and  program  training  for  experiment  support 
personnel  including  observers,  roleplayers,  data 
collectors,  M&S  support,  collection  managers  and 
others . 

2.6.3  Plan  and  program  training  for  experiment 
participants  to  ensure  familiarity  (as  appropriate) 
with  operational  concepts,  experiment  infrastructure, 
experiment  information  exchange  processes,  and  roles 
of  other  participants. 

2.7  Conduct  security  planning. 

2.7.1  Identify  security  issues  and/or  engineering 
requirements  using  appropriate  instructions,  e.g.. 
Defense  Information  Technology  Security  Certification 
and  Accrediation  Process  (DITSCAP)  or  National 
Information  Assurance  Certification  and  Accreditation 
Process  (NIACAP) . 

2.7.2  Assess  impact  of  security  requirements  or 
experiment  and  data  availability,  e.g.,  access  by 
foreign  nationals,  "uncleared"  researchers,  storage  of 
data  and  documents,  system  connectivity. 

2.7.3  If  needed,  secure  and  schedule  needed  security 
resources,  e.g.,  storage  containers,  destruction 
resources,  access  control  resources,  storage  devices, 
etc . 

2.8  Determine  risk  and  define  risk  management  procedures. 

2.8.1  Identify  types  and  levels  of  risk  to  experiment 

success,  infrastructure,  personnel  availability, 
funding,  schedule,  and  cost 
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2.8.2  Determine  requirements  and  options  to  mitigate 
risk . 

2.8.3  Develop  risk  mitigation  plans  for  high-risk 
elements  such  as  immature  infrastructure  components, 
use  of  surrogates,  variance  in  software/hardware 
versions,  long  lead  items,  and  generation  of  essential 
databases . 

2.8.4  Establish  guidance  on  documentation  requirements 
of  any  design  and  development. 

2.9  Conduct  schedule  planning  and  control. 

2.9.1  Ensure  that  the  proposed  schedule  is  viable  given 
the  scope  of  the  experiment,  the  identified  risks,  and 
the  proposed  funding. 

2.9.2  Identify  critical  event  dependencies  and  long- 
lead  items  for  key  activities. 

2.9.3  Schedule : 

Adequate  time  for  development  and  conduct  of  training 
for  observers,  support  staff,  as  well  as  participants. 
Progress  reviews  for  assessments  of  risk  and  progress 
for  experiment  participants,  through  all  phases  of  the 
experiment . 

Progress  reviews,  peer  reviews,  and  critical  decision 
reviews  for  programmatic  goals,  including  one  at  the 
completion  of  planning  phase. 

Sufficient  time  for  testing  and  integration  of  the 
hardware/ software/ infrastructure . 

2.10  Conduct  transition  planning. 

2.10.1  Develop  a  transition  plan  to  pass  findings  and 
conclusions  to  stakeholders. 

2.10.2  Implement  a  communications  strategy  to 
disseminate  information  to  participating  and 
interested  parties,  e.g.,  form/type  of  briefings,  web 
posting,  email  distribution  lists,  etc. 

2.10.3  Identify  budget  or  Program  Objective  Memorandum 
(POM)  submissions  potentially  affected  by  the 
experiment . 

2.10.4  Identify  and  schedule  a  knowledge  repository— 
e.g.,  an  archive  or  website— for  experimental  results. 

2.10.5  Ensure  agreement  and  funding  support  for  any 
residual  assets  either  left  in  place  or  transitioned 
to  the  appropriate  sponsors. 

2.10.6  Estimate  funding  requirements  for  follow-on 
research  and  development  activities,  e.g.,  refinement 
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of  prototype,  additional  experiments,  acquisitions, 
etc . 

3.  Development  and  Validation 

3.1  Complete  design  and  implementation  plan. 

3.1.1  Schedule  or  complete  final  experiment  design. 

3.1.2  Ensure  final  experiment  architecture  and 
implementation  are  on  track. 

3.1.3  Ensure  development  of  the  data  collection  plan  is 
on  schedule  or  complete,  including  quality  control 
processes . 

3.1.4  Ensure  test  and  integration  is  proceeding  on 
schedule,  or  completed. 

3.1.5  Schedule  and/or  complete  development  and  testing 
of  infrastructure,  support  tools,  required  databases. 

3.1.6  Ensure  experiment  analysis  and  evaluation  plan 
are  complete  and  all  activities  are  on  track. 

3.1.7  Establish  and  determine  application  of  measures 
of  effectiveness,  metrics,  and/or  success  criteria  and 
incorporate  in  the  analysis  and  evaluation  plan. 

3.1.8  For  experiment/research  campaigns,  establish 
iterations  and  entry/exit  criteria. 

3.1.9  If  required,  prepare  help  desk  for  the 
experiment . 

3.1.10  Ensure  security  policies  reviews  are  completed, 
accreditations  obtained,  or  formal  waivers  obtained. 

3.1.11  Ensure  that  development  and  implementation  of 
operational  scenarios,  concepts,  and  script  are 
complete  or  proceeding  on  schedule,  including 
Models  and  simulations. 

Training. 

Orientation  and  familiarization  plans  and  materials 
for  participants. 

3.2  Execute  readiness  review. 

3.2.1  Schedule  review  of  experiment  preparations  for 
the  research  sponsor (s) . 

3.2.2  Identify  a  means  to  address  problems  and  issues, 
and  ensure  availability  of  resources  for  corrective 
actions . 

Ensure  data  collection  instruments  and  participant 
forms/questionnaires  are  available. 

Schedule  observer  interviews . 
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Schedule  appropriate  mechanisms  to  capture  and 
disseminate  information  such  as  recording  of 
briefings . 

3.2.3  Implement  help  desk. 

3.2.4  Establish  visitor  access  and  control  procedures. 

3.2.5  Ensure  data  collection  team  and  resources  are 
ready,  including  a  means  to  archive  all  collected 
data . 

3.2.6  Implement  process  to  capture  Lessons  Learned. 

3.2.7  Implement  control  processes  for  changes  to 
experiment  environment,  infrastructure,  and  procedures 
that  may  be  caused  by  anomalous  interruptions. 

3.2.8  Describe  and  document  rehearsal  procedures  for 
all  participants  and  assign  all  responsibilities. 

3.2.9  Ensure  sufficient  time  is  included  in  the 
schedule  to  take  corrective  action  as  needed  after  the 
rehearsal . 

3.2.10  Ensure  all  activities  in  the  conduct  of  the 
experiment  are  addressed  and  documented,  including 
those  of  participants,  observers,  and  visitors,  in 
addition  to  equipment,  systems,  and  infrastructure. 

4.  Preparation  and  Rehearsal 

4 . 1  Conduct  rehearsal . 

4.1.1  Include  participants,  observers,  support 
personnel,  and  (acting)  visitors  engaged  in  the 
rehearsal . 

4.1.2  Exercise  the  supporting  infrastructure,  including 
instrumentation . 

4.1.3  Exercise  all  key  activities  and  processes  in  the 
experiment  design  and  plan-  including  data  collection 
and  analysis. 

4.1.4  Ensure  rehearsal  includes  practice  vignettes  and 
end-to-end  scenarios. 

4.1.5  Stress  the  system  architecture  to  ensure  all 
experiment  requirements  are  supported. 

4.1.6  Ensure  a  process  exists  to  capture  anomalies  and 
unexpected  disruptions  and  to  take  corrective  actions 
in  the  experiment. 

4.1.7  Verify  that  the  experiment  addresses  the  original 
ob j  ectives . 

5.  Execution 

5.1  Collect  data. 
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5.1.1 


Collect,  verify,  and  archive  data  including 
observer  notes  and  informal  interviews. 

5.1.2  Monitor  quality  control  mechanisms  for  data 
collection . 

5.2  Ensure  daily  communications  plan  is  followed. 

5.2.1  Schedule  roundtable  information  exchange  meetings 
of  the  data  collection  team  and  other  observer  groups. 

5.2.2  Archive  notes,  data  collection  instruments,  and 
other  materials  such  as  emails,  voice  communications 
logs  generated  during  experimentation  events. 

5.3  Document  experiment  process  and  lessons  learned. 

5.3.1  Document  changes  (functionality)  or  deviations 
from  experimentation  plan  in  accordance  with  technical 
control  plan. 

5.3.2  Record  down  time  and  perceived  impact  on 
experiment  in  light  of  system  or  process  failures  such 
as  power  outages,  interruptions,  and  loss  of 
participants  in  logs  and  data  compilation. 

5.3.3  Capture  and  document  lessons  learned  in 
experiment  process. 

5.3.4  Provide  "hot  wash"  briefing  and  discussion  for 
participants  and  capture  resulting  comments  and 
observations . 

6.  Analyses,  Evaluation,  and  Transition 

6.1  Provide  a  "quick  look"  report  and  briefing  for 
stakeholders . 

6.1.1  Report,  at  a  minimum,  the  assumptions  and  major 
findings  of  the  experiment  and  any  initial 
recommendations  affecting  the  experiment  campaign  or 
related  operations. 

6.1.2  Identify  any  unresolved  issues,  uncertainties, 
and  sensitivities  discovered  in  the  experiment. 

6.1.3  Provide  for  the  widest  possible  circulation  of 
the  briefing  and  report  and  invite  review  and 
critique . 

6.1.4  Caveat  "quick  reaction"  results  to  avoid 
premature  conclusions  and  recommendations. 
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6.2  Prepare  and  distribute  a  report  of  preliminary 

findings  within  a  reasonably  short  time  after  the 
experiment . 

6.2.1  Distinguish  clearly  between  findings  (factual 
observations  and  data)  and  interpretations  of  the 
results . 

6.2.2  Describe  the  effects  of  interruptions, 
disruptions,  anomalies,  etc. 

6.2.3  Collect  peer  review  results  and  incorporate  them 
into  revised  and  future  reports. 

6.3  Prepare  and  publish  formal  reports. 

6.3.1  Adopt  and  implement  a  publication  and 
dissemination  plan  to  provide  a  range  of  products 
(decision  papers,  summary  reports,  briefings, 
scientific  papers,  articles,  and  books)  suited  to  the 
needs  of  various  audiences  and  stakeholders. 

6.3.2  If  appropriate,  provide  a  synthesis  of  findings 
and  interpretations  from  across  several  related 
experiments,  especially  those  in  the  experimentation 
campaign  that  includes  the  present  experiment. 

6.3.3  Provide  recommendations  for  iterations  of  the 
experiment  or  practical  application  of  results  in 
exercises,  as  well  as  future  experiments  based  on 
issues  uncovered  or  knowledge  derived. 

6.3.4  Provide  lessons  learned  about  the  experimentation 
processes,  tools  used,  and  infrastructure.  Incorporate 
considerations  for  scaling  or  expanding  results  by 
further  experimentation,  modeling  and  simulation,  or 
other  means . 

6.3.5  Identify  and  address  programming  and  budgeting 
implications,  including  resources  needed  to  transition 
results,  e.g.,  refinement  of  prototypes,  more 
experiments,  implications  for  on-going  acquisitions. 
Identify  any  budget  or  POM  submissions  affected. 


6.4  Archive  experiment  design,  data,  and  results  for 
future  use. 

6.4.1  Collect  all  data  records,  interview  transcripts, 

scenario  descriptions,  training  manuals,  and  other 
artifacts  immediately  after  execution  of  the 
experiment  and  preserve  them  in  their  original  form. 
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6.4.2 


Compile  and  archive  a  dictionary/glossary  of  the 
terms,  constructs,  and  acronyms  used  in  the 
experiment . 

6.4.3  Compile  and  archive  a  dictionary  of  methodology 

and  metrics  (including  definitions  and  scales)  used  in 
the  experiment. 
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COMFLTFORCOM  INSTRUCTION  3900. 1A 


CFFCINST  3900. 1A 
N8 

22  Dec  03 

COMFLTFORCOM  INSTRUCTION  3900 ■ 1A 

Subj :  SEA  TRIAL 

Ref:  (a)  Naval  Power  21 

(b)  CNO  011515Z  Oct  01  {Sea  Power  21  Implementation) 

Enel:  (1)  Sea  Trial  process  diagram 

1.  Purpose .  To  provide  Commander,  U.S.  Fleet  Forces  Command 
(CFFC)  policy  guidance,  establish  responsibilities,  and 
prescribe  general  procedures  for  establishing  and  executing  Sea 
Trial  in  support  of  Sea  Power  21. 

2.  Cancellation:  COMFLTFORCOMINST  3900.1 

3.  Background .  Naval  Power  21,  reference  (a),  provides 
SECNAV's  vision  for  Navy  and  Marine  forces.  CNO  and  CMC  have 
established  Sea  Power  21  and  Marine  Corps  21  as  the  Navy  and 
Marine  Corps'  strategy  for  the  21st  Century.  Sea  Power  21  is 
built  around  the  core  concepts  of  Sea  Strike,  Sea  Shield,  Sea 
Basing,  enabled  by  FORCEnet ,  with  US  Marine  Corps  Expeditionary 
Maneuver  Warfare  (EMW)  and  Joint  maneuver  warfare  from  the  Sea 
Base  fully  embedded.  Sea  Trial  will  integrate  emergent  concepts 
and  technologies  into  an  experimentation  process  that  will 
produce  continuous  improvements  in  naval  warfighting.  Reference 
(b)  established  CFFC  as  the  lead  agent  for  Sea  Trial. 

4.  Objectives.  Through  Sea  Trial,  Fleet  Forces  Command  (FFC ) 
will  formalize  and  fully  integrate  concept  development  and 
technology  insertion  into  the  experimentation  process.  Sea 
Trial  will  be  Naval  in  focus  through  close  coordination  with  the 
Marine  Corps  Combat /Force  Development  Process,  and  will  support 
concurrent  development  and  testing  of  joint  warfare  capabilities 
from  the  Sea  Base.  A  comprehensive  Sea  Trial  Concept 
Development  and  Experimentation  (CD&E)  Plan  will  rapidly  mature 
Sea  Shield,  Sea  Strike,  Sea  Basing  and  FORCEnet  concepts  and 
technologies,  codify  experiment  results  in  doctrine,  and  reflect 
in  programs  of  record.  The  CD&E  Plan  will  be  concept-based  and 
aligned  to  mission  capabilities  gaps  and  Fleet  priorities. 
Emphasis  will  be  on: 
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a.  Speed  --  Expeditious  induction  of  new  concepts  and 
technologies  into  the  experimentation  process,  along  with  rapid 
transition  of  promising  outcomes  into  doctrine  and  program. 

b.  Comprehensive  --  Single  point  visibility  over  all  Navy 
experimentation  other  than  the  Discovery  and  Invention  portion 
of  the  Navy's  Science  and  Technology  effort. 

c.  Jointness  --  Maximum  incorporation  of  Joint  Concept 
Development  and  Experimentation  (JCD&E)  into  Navy  processes. 

d.  Relevance  --  Applicability  of  experimentation  to  mission 
capability  gaps  in  naval  warfighting. 

e.  Economy  --  Maximum  effectiveness  and  efficiency  from 
Concept  Development  and  Experimentation  (CD&E)  efforts. 

5.  Joint  Applicability.  Marine  Corps  operating  forces  are  part 
of  the  fleet.  Accordingly,  the  Marine  Corps  will  be  an  equal 
partner  at  each  level  within  the  Sea  Trial  process  and  will  use 
the  Sea  Trial  process  to  coordinate  their  (CD&E)  efforts  that 
are  inherently  naval  and  not  service  specific.  As  the  Deputy 
Commandant  for  Combat  Development,  Commanding  General  Marine 
Corps  Combat  Development  Command  (CG  MCCDC)  is  the  Marine  Corps' 
Sea  Trial  coordinator.  Sea  based  operations  will  be 
increasingly  important  to  overall  j oint/combined  warfare 
missions.  To  that  end,  coordinated  efforts  with  the  Army,  the 
Air  Force  and  other  forces  are  important  to  maximize  the  payoff 
of  ongoing  naval  force  experimentation  to  future  joint /combined 
war  fighting  effectiveness.  Concurrently,  and  to  the  greatest 
extent  possible.  Sea  Trial  will  also  support  future  coalition 
operations.  Finally,  Joint  Forces  Command  {JFCOM)  plays  a  key 
role  in  JCD&E,  and  a  strong  relationship  will  exist  between  Sea 
Trial  efforts  and  JFCOM  interests. 

6.  Sea  Trial  Organization.  Sea  Trial  is  a  CFFC-led 
collaborative  effort  that  includes  COMPACFLT,  COMNAVEUR,  DCNO 
(N6/N7),  CNR,  CNVJDC ,  CG  MCCDC  (DC,  CD),  C3l||  C2F,  C5F,  C6F,  C7F, 
NETWARCOM  and  others  involved  in  Naval  CD&E 

a.  Sea  Trial  Executive  Steering  Group  (STESG) .  CFFC  will 
establish  and  chair  a  flag  officer  Sea  Trial  Executive  Steering 
Group  (STESG)  comprised  of  COMPACFLT,  COMUSNAVEUR,  DCNO  (N6/N7), 
CNR,  CNWDC  and  CG  MCCDC  (DC,  CD)  .  COMSECONDFLT ,  partnered  with 
COMF1 FTHFLT  and  COMSIXTHFLT,  and  COMTHIRDFLT,  partnered  with 
COMSEVENTHFLT ,  will  participate  as  Sea  Trial  Operational  Agents 
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for  Sea  Strike,  Sea  Basing  and  Sea  Shield.  COMNAVNETWARCOM  will 
participate  as  the  Sea  Trial  Operational  Agent  for  FORCEnet . 
Marine  Corps  operational  forces  (MARFORLANT  and  MARFORPAC) 
designated  by  CG  MCCDC  (DC, CD)  will  participate  as  counterparts 
to  the  Operational  Agents  to  address  Naval  issues  as 
appropriate.  A  single  Flag-level  representative  of  each  SYSCOM 
will  participate  in  STESG  meetings  as  advisory  members  for 
matters  relating  to  Research  and  Development  (R&D)  and 
industrial  base  capability.  A  COMOPTEVFOR  flag  level 
representative  will  participate  in  STESG  meetings  and  advise  on 
Operational  Test  and  Evaluation  (OT&E)  matters.  The  STESG  will 
coordinate  an  agile,  responsive  and  transformational  Sea  Trial 
process .  Members  and  designated  commands  will  coordinate 
quarterly  to  validate  recently  completed  military  utility 
assessments,  determine  if  additional  testing  is  required  and/or 
recommend  follow-on  action  including  analysis  and  programming 
via  the  Naval  Capabilities  Development  Process  and  PPBE,  approve 
the  Sea  Trial  Execution  Plan  including  any  modifications, 
additions  or  deletions  of  follow-on  events  based  on  the  results 
of  other  completed  testing,  resolve  issues  of  resources  and/or 
priorities,  assess  non-traditional  solutions  to  ongoing 
warfighting  development  issues,  and  make  recommendations  to  CFFC 
on  the  viability  of  emergent  concepts  and  technologies.  E/vents 
approved  by  the  Operational  Agent  but  remaining  unscheduled  will 
be  reviewed  at  each  STESG.  This  dynamic  review  process  will 
ensure  Sea  Trial  remains  agile  and  responsive  to  emerging 
concepts  or  materiel  solutions  to  war fighting  challenges  - 

b.  Operational  Agents  (OA) ■  COMSECONDFLT  (partnered  with 
COMF1 FTHFLT  and  COMSIXTHFLT)  is  the  Sea  Trial  Operational  Agent 
for  Sea  Strike  and  Sea  Basing.  COMTHIRDFLT  (partnered  with 
COMSEVENTHFLT )  is  the  Sea  Trial  Operational  Agent  for  Sea 
Shield.  COMNAVNETWARCOM  is  the  Sea  Trial  Operational  Agent  for 
FORCEnet.  Operational  Agents  are  responsible  for  overall 
prioritization  and  coordination  of  all  aspects  of  warfighting 
CD&E  within  their  respective  pillar  areas.  The  Operational 
Agents,  supported  by  members  of  the  Fleet  Collaborative  Teams 
(FCT)  and  NWDC,  will  validate  proposed  CD&E  initiatives 
(contained  in  the  Sea  Trial  Information  Management  System 
(STIMS) ) ;  oversee  the  planning,  coordination  and  conduct  of  Sea 
Trial  events;  and  brief  results  to  the  STESG.  Any  activities 
desiring  to  conduct  a  Sea  Trial  event  that  falls  in  part  or 
wholly  under  one  or  more  of  these  pillars  will  fully  coordinate 
such  initiatives  through  the  respective  OA,  and  will  ensure 
these  initiatives  are  entered  into  the  STIMS  database.  Marine 
Corps  operational  forces  (MARFORLANT  AND  MARFORPAC)  designated 
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by  CG  MCCDC  (DC, CD)  will  participate  as  counterparts  to  the 
Operational  Agents  to  address  Naval  issues  as  appropriate. 

c ,  Fleet  Collaborative  Teams  (FCTs)  -  These  teams  are 
chartered  by  CFFC  and  will  contribute  to  the  Naval  Capability 
Development  Process  (NCDP)  and  CD&E  efforts.  FCTs  will  leverage 
the  intellectual  capital  of  Warfare  Centers  of  Excellence 
(WCOE),  NWDC,  ONR,  TYCOMs ,  SYSCOMs ,  Fleets  and  other  commands, 
and  are  generally  organized  along  Mission  Capability  Package 
(MCP)  lines  under  the  Naval  Capability  Pillars  (NCP)  of  Sea 
Strike,  Sea  Shield,  Sea  Basing  and  FORCEnet .  FCTs  will  provide 
the  Operational  Agents  with  the  necessary  expertise  to  develop 
and  evolve  the  Sea  Power  21  operating  concepts.  WCOEs  are  key 
to  the  CD&E  aspect  of  the  FCT's  charter,  as  they  will  be 
responsible  for  key  elements  of  doctrine,  tactics,  techniques 
and  procedures  (TTP) ,  Lessons  Learned  Remedial  Action,  and  CD&E. 
This  directive  authorizes  Operational  Agents,  acting  on  behalf 
of  CFFC,  to  direct  FCTs  in  their  MCP  areas  for  development  and 
implementation  of  experimentation  plans.  In  addition  to  their 
role  in  the  NCDP  process,  FCTs  will  contribute  to  CD&E  in  the 
following  manner: 

(1)  Develop  Sea  Power  21  Operating  Concepts.  The  FCTs 
are  responsible  to  their  respective  Operational  Agents  for 
development  and  evolution  of  Sea  Power  21  operating  concepts  and 
the  overall  direction  for  CD&E  in  their  respective  areas . 

NWDC  will  coordinate  this  effort  across  all  OA  Sea  Trial 
activities,  and  FFC  N8  will  assist  in  this  process. 

(2)  Provide  Subject  Matter  Experts  (SMEs)  for,  and 
participate  in,  concept  development  initiatives  germane  to  their 
mission  area  to  include  wargaming  (including  modeling  and 
simulation),  MCP  Integrated  Product  Teams  (IPT) ,  Sea  Trial 
Military  Utility  Assessment  (MUA)  teams,  limited  objective 
experiments  (LOEs),  employment  of  operational  prototypes  and 
major  field  experiments  (e.g.  Fleet  Battle  Experiments  (FBEs) , 
and  USMC  Advanced  Warfighting  Experiments  (AWEs) ) .  Operational 
Agents  will  periodically  convene  concept  development  workshops 
that  combine  the  efforts  of  SMEs,  operators,  technologists,  and 
network  specialists  to  formulate  innovative  new  operating 
concepts  that  harness  advanced  systems  to  provide  enhanced 
capabilities  in  their  assigned  mission  areas. 

(3)  In  cooperation  with  and  with  the  approval  of  the 
OA ( s ) ,  develop  Sea  Trial  Execution  Plans  that  support  the  Sea 
Trial  CD&E  Plan.  FCTs  will  develop  Execution  plans  for  their 
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mission/functional  area  for  approval  by  the  Operational  Agent  in 
coordination  with  CFFC  (NWDC  facilitating) . 

(4)  Ensure  Sea  Trial  Events  are  properly  planned  and 
scheduled.  The  FCT,  assisted  by  NWDC,  will  identify 
requirements  necessary  for  Fleet  schedulers /planners . 

(5)  Review  and  Process  Military  Utility  Assessments. 

(6)  Participate  in  TACMEMO  development  and  support 
experimentation  and  Tactics,  Techniques  and  Procedures  (TTP)  for 
validated  capabilities. 

(7)  Oversee  and  report  to  their  respective  Operational 
Agents  on  the  implementation  of  the  approved  and  funded  CD&E 
activities  laid  out  in  the  Sea  Trial  Execution  Plan. 

7 .  Sea  Trial  Process . 


a.  CD&E  initiatives,  whether  concept  or  technology  based, 
will  be  submitted  by  the  FCTs,  OPNAV,  ONR,  SYSCOMS,  Industry  or 
other  agencies  to  NWDC  for  consolidation,  tracking,  and 
coordination  via  the  STIMS  database.  Demonstrations,  events, 
experiments,  or  prototypes  designed  to  explore  new  technologies 
or  concepts  with  the  fleet,  or  on  behalf  of  the  fleet,  require 
strict  adherence  to  the  Sea  Trial  process.  This  includes 
entering  information  into  STIMS.  NWDC  will  ensure 
distribution  to  all  activities. 

b.  The  OA,  assisted  by  NWDC  and  FCTs,  will  examine  CD&E 
initiatives  to  determine  their  technical  feasibility,  relevance 
to  the  Sea  Trial  CD&E  Plan,  whether  the  initiative  meets  Fleet 
priorities  (as  determined  by  OAs)  and/or  Mission  Capability  gaps, 
and  identify  the  appropriate  experimental  venue.  Initiatives 
failing  to  meet  the  criteria  will  be  rejected  by  the  Operational 
Agent  and  the  sponsor  and  CFFC  will  be  informed  of  the  rationale. 

c.  The  Operational  Agent  will  examine  validated  initiatives 
to  determine  if  additional  monetary  resources  are  required. 

(1)  Validated  initiatives  that  are  adequately  resourced 
will  continue  to  be  processed  as  follows.  The  event's  sponsor, 
in  coordination  with  the  FCTs  and  NVJDC,  will  develop  an 
experimentation  plan  and  submit  it  to  the  OA  for  approval.  This 
plan  will  describe  how  the  experiment  will  be  conducted  and  will 
include  a  data  collection  and  analysis  plan  (DCAP)  with 
appropriate  metrics.  Once  approved,  these  validated  initiatives 
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will  become  Sea  Trial  events  and  the  OA  will  provide  an 
information  brief  to  the  STESG  at  the  first  opportunity.  CFFC 
(with  NWDC  as  agent)  will  then  assign  the  Military  Utility 
Assessment  (MUA)  team  and  MUA  timeline. 

(2)  Initiatives  with  resource  issues  will  be  forwarded 
to  NWDC  for  broad  effort  resolution,  or  (if  beyond  existing 
resource  capability)  for  presentation  by  NWDC  to  the  next  STESG 
for  resolution  (i.e.  funding,  postponement,  modification,  or 
termination) .  In  those  instances  where  an  important  Sea  Trial 
opportunity  will  be  missed  by  waiting  for  the  next  full  STESG 
meeting,  NWDC  will  prepare  and  immediately  electronically 
forward  to  all  STESG  principals  a  summary  of  the  proposed  event 
and  actions  required  to  support  a  coordinated  decision,  ensuring 
the  responsiveness  of  the  overall  system. 

d.  NWDC  will  align  all  events  across  all  Naval  Capability 
Pillars  and  present  a  consolidated  Execution  Plan  to  the  STESG. 
The  Execution  Plan  will  be  presented  to  the  STESG  in  June  and 
will  outline  the  CD&E  intentions  for  the  subsequent  two  fiscal 
years.  Changes  to  the  Execution  Plan  will  be  briefed  to  the 
STESG  by  the  OA  proposing  the  change. 

e.  The  OA  will  publish  a  Quick  Look  upon  completion  of  the 
Sea  Trial  event  and  provide  it  to  the  STESG. 

f.  The  MUA  team  will  complete  the  MUA  in  accordance  with  the 
MUA  timeline  and  submit  it  to  the  OA  for  approval.  Completed 
MUAs  will  be  distributed  and  briefed  to  the  STESG  by  the 
respective  OA.  NWDC  will  coordinate  these  briefings  for 

the  STESG,  with  the  goal  of  providing  an  efficient  summary  of 
Sea  Trial  products  completed  since  the  previous  STESG  meeting, 
and  as  a  basis  for  review  of  planned  and  pending  Sea  Trial 
events  in  support  of  dynamic  validation  of  the  overall  Sea  Trial 
CD&E  Plan.  MUA  recommendations  will  be  categorized  into 
Doctrine,  Organization,  Training,  Materiel,  Leadership, 

Personnel  and  Facilities  format.  The  STESG  will  rate  each 
recommendation  based  on  four  possible  outcomes.  OPNAV  N6/N7  is 
the  single  point  of  entry  for  FFC  products  (with  the  exception 
of  doctrine)  resulting  from  Sea  Trial  experiments.  Events 
classified  as  having  significant  enhanced  warfighting  capability 
impact  will  be  presented  to  CFFC  for  immediate  delivery  to  OPNAV 
N6/N7  for  insertion  into  the  NCDP  or  forwarded  by  N6/N7  to  the 
applicable  OPNAV  process  agent  for  action  (e.g.  Nl  (Manpower  and 
personnel),  N4  (Readiness,  material,  facilities),  NOOT 
(Training) ) .  Those  items  that  are  doctrinal  related  will  be 
passed  by  CFFC  directly  to  NWDC.  Events  classified  as  having 
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positive  enhanced  warfighting  capability  impact  will  be 
presented  to  CFFC  for  delivery  to  OPNAV  N6/7  and  distributed  as 
mentioned  above.  CFFC  will  carefully  track  the  progress  of 
these  inputs.  Events  evaluated  as  not  mature  enough  and 
requiring  further  examination  will  be  rescheduled  for  additional 
experimentation  as  warranted.  Events  evaluated  as  not  providing 
an  impact  will  be  closed  out. 

g.  The  status  of  all  Sea  Trial  events  will  be  maintained  on 
STIMS  and  will  be  available  for  stakeholder  review.  This  status 
will  be  provided  to  the  STESG  as  a  read  ahead  before  meetings 

8 .  Sea  Trial  Concept  Development  and  Experimentation  (CD&E> 

Plan .  The  Sea  Trial  CD&E  Plan  will  be  directed  by  CFFC  and 
drafted  by  NWDC  based  upon  the  inputs  of  the  Operational  Agents. 
The  CD&E  Plan  will  be  focused  10  years  out  and  contain  a  detailed 
listing  of  Fleet  priorities  and  mission  capability  gaps .  This 
CD&E  Plan  will  be  a  dynamic  document,  that  will  establish  metrics 
to  assist  OAs  in  determining  applicability  of  their  Sea  Trial 
efforts  and  allow  the  STESG  to  measure  overall  Sea  Trial 
progress,  support  proper  planning  for  complex  testing 
requirements,  and  take  full  and  increased  advantage  of  emergent, 
smaller  scale,  more  focused  testing  opportunities.  The  CD&E  Plan 
should  implement  a  comprehensive  Sea  Trial  Roadmap  that 
integrates  studies,  war  games,  experimentation,  and  exercises 
with  proposed  evaluation  metrics.  This  CD&E  Plan  should  include 
those  promising  concepts  and  technologies  that  speed  Sea  Power  21 
capabilities  to  the  Fleet. 

9.  Sea  Trial  Execution  Plan.  The  Sea  Trial  Execution  Plan  is  a 
two-year  document  and  represents  a  collection  (synchronized 
across  NCPs)  of  OA  planned  Sea  Trial  events.  The  document  will 
be  prepared  by  NWDC  based  on  inputs  from  the  OAs  and  briefed  to 
the  STESG  for  approval  each  June.  The  Execution  Plan  will  also 
contain  evaluation  metrics  and  an  execution  timeline. 

Subsequent  changes  in  OA  planning  requiring  a  change  to  the 
Execution  Plan  will  be  briefed  by  the  individual  OA. 

10.  Sea  Trial  Information  Management  System  (STIMS) .  STIMS,  an 
NWDC  developed  and  maintained  interactive  web  based  database, 
will  be  the  central  library  of  initiatives  and  technologies  and 
will  be  used  as  the  tool  to  manage  Sea  Trial  events  and  related 
activity.  Operational  Agents,  supported  by  all  FCTs  and  all 
other  Sea  Trial  event  sponsors  are  responsible  for  populating 
and  updating  all  STIMS  database  entries  for  their  respective 
efforts.  Other  commands,  including  CNR,  OPNAV,  SYSCOMS,  etc.. 
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will  ensure  all  experimentation  is  fully  documented  in  SUMS. 
The  initiative  submission  format  will  be  contained  in  STIMS 
and  entry  into  STIMS  will  be  the  minimum  essential  requirement 
for  any  event  to  be  considered  for  future  Sea  Trial  status . 

NWDC  will  maintain  business  rules  that  will  be  used  by  CFFC  for 
utilization  of  STIMS  by  all  stakeholders. 

11.  External  Sea  Trial  Interfaces.  Sea  Trial  must  be 
synchronized  with  other  processes  related  to  CD&E  .  CFFC  (with 
NWDC  as  agent)  shall  work  with  OPNAV,  ONR,  USJFCOM  and  other 
services /agencies  responsible  for  these  processes  to  create  a 
fully  integrated  approach  to  Naval  transformation  in  support  of 
j oint /combined  warfighting  capabilities  development. 

12 .  Roles  and  Responsibilities. 

a.  Commander,  U-S.  Fleet  Forces  Command. 


(1)  Is  the  Lead  Agent  for  Sea  Trial  and  the  supported 
Commander  for  all  Sea  Strike,  Sea  Basing,  Sea  Shield,  and 
FORCEnet  CD&E  under  Sea  Trial . 

(2)  Is  responsible  for  the  composition  of  Sea  Trial  CD&E 
Plan  (supported  by  NWDC) ,  review,  promulgation  and  periodic 
modification  following  STESG  deliberations.  The  CD&E  Plan  is 
the  principle  definitive  tool  for  establishing  Sea  Trial  goals 
and  will  be  the  basis  which  overall  Sea  Trial  measures  progess. 

(3)  Will  chair  the  STESG  and  act  as  the  approval 
authority  for  STESG  recommendations  . 

(4)  Will  (with  NWDC  as  agent)  assign  the  MUA  team  and 
associated  timeline. 

(5)  Will  endorse  Sea  Trial  results  and  direct 
implementation  via  OPNAV  NCDP  and  PPBE  processes. 

(6)  FFC  N82  is  the  FFC  staff  project  office  for  Sea 
Trial.  FFC  N80  is  the  overall  coordinating  authority  for  FCTs, 
with  FFC  N82  providing  assistance  for  CD&E  efforts. 

b.  Commander,  U.S.  Pacific  Fl  eet/ Commander,  U.S.  Naval 
Forces  Europe. 


(1)  Provide  forces  when  available  to  support  the  Sea 
Trial  Execution  Plan. 
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(2)  Integrate  experiments  into  fleet  exercises  to  the 
maximum  extent  possible  while  minimizing  the  impact  on  fleet 
readiness  and  training. 

(3)  Recommend  pertinent/relevant  emergent  technologies 
and  concepts  as  Sea  Trial  events.  Facilitate  efforts  to  fast 
track  select  technologies /concepts  based  on  Sea  Trial 
assessments . 

(4)  Identify  to  the  Operational  Agents  focus  areas  for 
prioritizing  CD&E. 

(5)  Provide  representation  to  the  STESG . 
c .  Navy  Warfare  Development  Command  (NWDC) ■ 

(1)  Is  the  supporting  commander  as  the  Sea  Trial  Project 
Coordinator  across  all  Navy/Naval  sea  trial  activities.  NWDC  is 
the  supported  commander  for  development  of  the  Sea  Trial  CD&E 
Plan  and  maintenance  of  the  STIMS  website.  The  CD&E  Plan  and/or 
the  Execution  Plan  may  be  modified/ updated  following  each  STESG 
meeting.  CFFC  may  direct  an  urgent  change  at  any  time. 

(2)  Will  link  proposed  Sea  Trial  events  to  warfighting 
capabilities  required  to  support  Sea  Power  21. 

(3)  will,  through  representation  on  all  FCTs ,  coordinate 
with  PEOs,  SYSCOMs,  WCOEs ,  T&E  Centers,  labs  and  ONR  to  develop 
an  integrated  and  synchronized  CD&EI  Plan  that  focuses  on  Fleet 
priorities  and  Mission  Capability  gaps.  The  OA  approved  execution 
plans  will  be  synchronized  by  NWDC  and  included  in  the  Sea  Trial 
Execution  Plan.  NWDC,  reporting  to  the  STESG,  will  coordinate 
these  efforts  across  all  OAs,  minimizing  (with  a  goal  of 
eliminating)  redundancies  in  experimentation  planning,  and 
providing  an  independent  assessment  of  the  combined  Sea  Trial 
process  in  addressing  MCP  gaps,  ancl  other  naval  and  joint 
warfighting  capability  development  requirements. 

(4)  Identify  for  CFFC,  via  the  STESG,  areas  that  are  not 
being  addressed  by  any  proposed  Sea  Trial  events  following 
assessment  of  the  overall  Sea  Trial,  effort . 

(5)  Focus  the  CD&E  Plan  in  accordance  with  CFFC, 
Operational  Agent,  OPNAV  N6/N7,  and  CG  MCCDC  priorities  while 
consistently  working  fco  meet  joint  requirements.  Ensure  the 
plan  remains  agile  and  aggressively  takes  advantage  of  rapidly 


CFFCINST  3900. 1A 
N8 

changing  testing  requirements  and/or  opportunities  to  rapidly 
advance  warfighting  capability  development  and  deployment. 

(6)  As  part  of  the  Sea  Trial  CD&E  plan  development, 
ensure  the  rapid  integration  of  lessons  learned,  fleet  feedback, 
and  previous  experimentation  analysis  and  assessment. 

(7)  Coordinate  and  liaison  with  CG  MCCDC  to  integrate 
(where  appropriate)  USMC  CD&E  into  Sea  Trial. 

(8)  Supported  by  the  Operational  Agents,  FCTs ,  and 
WCOEs,  plan  and  coordinate  execution,  analysis  and  assessment  of 
Fleet  Battle  Experiments  (FBEs ) ,  selected  limited  objective 
experiments  and  other  experiments  as  required. 

(9)  Promulgate  and  maintain  an  experimentation  planning 
guidebook  to  provide  execution  guidance  for  field  experiments. 

(10)  Synchronize  experimentation  activities  and  efforts 
to  co-evolve  the  technology,  doctrine/TTP  and  organizational 
changes  required  to  deliver  a  fielded  capability.  Recommend  to 
CFFC,  OA(s)  and  all  other  appropriate  commands  opportunities  to 
modify  or  cancel  planned  Sea  Trial  events  to  eliminate 
duplication  of  effort,  hold  down  cost,  and/or  free  assets  to 
focus  on  higher  priority  Sea  Trial  requirements. 

(11)  Leverage  other  service,  joint,  and  combined 
experimental  activities  to  the  maximum  extent  possible  and 
coordinate  the  fleet's  participation  in  these  other  experimental 
activities . 

(12)  Coordinate  all  Sea  Trial  Wargames/spiral  plans  with 
NWC  and  modify  them  as  required  to  support  Sea  Trial  CD&E  Plan 
priorities . 

(13)  Integrate  analysis  and  assessment  results  into 
DOTMLPF  change  recommendations  for  CFFC. 

(14)  As  operational  concepts  are  validated,  promulgate 
them  as  doctrine/TTP. 

(15)  Integrate  all  tac  d&e  activities  applicable  to  the 
Sea  Trial  CD&E  Plan. 

(16)  Develop  and  host  the  interactive  STIMS  database  and 
provide  assistance  to  activities  entering  data  into  STIMS  as 
necessary.  Report  to  CFFC  and  all  OA(s)  on  the  status  of  STIMS 
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usage,  to  ensure  all  Sea  Trial  events  are  fully  documented  in 
the  database. 

(17)  Coordinate  with  ONR,  MCCDC/VCNR  and  SYSCOMs  to 
ensure  R&D  activities  support  near,  mid  and  long  term  Sea  Trial 
priorities.  Obtain  visibility  over  ONR  Future  Naval 
Capabilities  (FNC)  and  Naval  Systems  innovations  (NSI)  to  assist 
in  determining  which  elements  are  mature  enough  for  inclusion  in 
Sea  Trial  events. 

(18)  Apprise  CFFC,  supported  by  the  Operational  Agents 
and  all  other  appropriate  commands,  of  guiding  requirements  and 
modifications/limitations  to  approved  STESG  plans  as  available 
funding  and/or  cost  estimates  to  support  events  evolves. 

(19)  Maintain  a  repository  of  Sea  Trial  results. 

d.  Commander,  Second  Fleet  (COMSECONDFLT )  Commander,  Third 
Fleet  ( COMTHIRDFLT )  and  Commander,  Naval  Network  Warfare  Command 
(COMNAVNETWARCOM)  . 

(1)  Serve  as  the  supported  Commander  and  Operational 
Agent  for  Sea  Trial  activities  associated  with  the  pillars  under 
their  cognizance  and  as  a  force  provider  for  Sea  Trial 
activities . 

(2)  Coordinate  fleet  activities  in  their  respective  AORs 
to  support  the  Sea  Trial  Execution  Plan . 

(3)  Coordinate  with  Marine  forces  and  other  numbered 
fleet  commanders  as  appropriate. 

(4)  Develop  Fleet  priorities  for  CD&E  and  nominate  high 
impact  capabilities,  activities  and  technologies  to  be 
incorporated  into  CD&E  plans. 

(5)  Review  and  approve  recommended  execution  plans 
developed  by  FCTs  in  support  of  assigned  pillars.  Forward  and 
brief  issues  to  the  STESG  for  resolution  as  required.  All 
designated  FCTs  will  fully  support  OAs  in  these  efforts. 

(6)  Support  joint  and  combined  CD&E  for  all  pillar 
related  activities,  as  directed,  including  allied  and  coalition 
requirements . 
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(7)  Provide  Military  Utility  Assessments  of  Sea  Trial 
experimentation  to  CFFC  for  approval  and  forwarding  as 
appropriate . 

(8)  COMTHIRDFLT  coordinate  Sea  Based  Battle  Lab  ( SBBL) 
activities  as  required  to  support  the  Sea  Trial  CD&E  Plan. 

e .  Naval  war  College  (HWC) ■ 


(1)  Coordinate  with  NWDC  to  fully  integrate  and 
synchronize  all  Sea  Trial  Wargames/ spiral  events  with  the  Sea 
Trial  CD&E  Plan. 

(2)  Coordinate,  with  NWDC,  a  long-term  wargaming  plan 
for  integration  into  the  Sea  Trial  CD&E  Plan. 

(3)  Coordinate  with  CG  MCCDC  for  integration  of  USMC 
CD&E  into  Sea  Trial  iDIRLAUTH  with  Expeditionary  Force 
Development  Center  (EFDC)  for  USMC  concept  development  and 
JCD&E.  DIRLAUTH  with  Marine  Corps  Warfighting  Lab  (MCWL)  for 
experimentation) . 

(4)  Coordinate  with  other  services  and  JFCOM  to  ensure 
Sea  Trial  initiatives  are  reflected  in  appropriate  ongoing  joint 
experimentation,  and  that  other  service  initiatives  are 
reflected  in  applicable  naval  wargaming. 


f.  Chief  of  Naval  Research  (CNR)  ■ 


(1)  Serve  as  the  supporting  commander  for  Sea  Trial  S&T 
activities . 

(2)  In  cooperation  with  NWDC  and  MCCDC,  nominate  FNC  and 
NSI  activities  that  are  sufficiently  mature  for  inclusion  in  Sea 
Trial . 

(3)  Provide  representation  to  the  STESG. 

(4)  Provide  representation  to  the  FCTs  as  assigned. 


g .  OPNAV  N6/N7 . 

(1)  Serve  as  supported  Warfare  Sponsor  for  Sea  Shield, 
Sea  Strike,  Sea  Basing,  and  FORCEnet  in  the  development  of 
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Mission  Capability  Packages  (MCP)  /Naval  Capabilities  Plans  and 
follow-on  development  of  a  Sponsor  Program  Proposal  (SPP) . 

(2)  As  co-chair  of  the  Technology  Oversight  Group  (TOG), 
and  working  with  ONR,  N6/N7  will  ensure  FNC  projects  support  the 
Sea  Trial  CD&E  Plan  objectives  where  applicable. 

(3)  Serve  as  supporting  Resource  Sponsor  for  Sea  Shield, 
Sea  Basing,  Sea  Strike  and  FORCEnet . 

(4)  Assign  Platform  Sponsors  as  contributing  Sea  Trial 
resource  sources  for  addressing  capability  gaps  identified  by 
both  the  NCDP  and  as  determined  by  Operational  Agents  {Working 
together  with  OPNAV  Warfare  Sponsors  in  a  collaborative  role) . 

(5)  Incorporate  Sea  Trial  results  as  endorsed  by  CFFC 
into  the  NCDP  or  forward  to  the  applicable  OPNAV  process  agent 
for  action  (e.g.  N1  (Manpower  and  Personnel),  N4  (Readiness, 
Material,  Facilities) ,  N00T  (Training) >  .  Those  items  that  are 
doctrine-related  will  be  passed  by  CFFC  directly  to  NWDC . 

(6)  Provide  representation  to  the  STESG . 

(7)  Support  the  FCTs  through  the  consulting  membership 
of  the  various  MCP  leads  to  provide  a  conduit  of  information 
from  the  MCPs  and  NCPs . 

h.  Warfare  Centers  of  Excellence  (WCOE) . 


(1)  Integrate  tactical  development  and  evaluation  into 
the  Sea  Trial  CD&E  Plan  to  include  TTP  development  for 
experiments,  TACMEMO  development  for  use  of  operational 
prototype  systems,  and  schedule  events  through  NWDC  in 
conjunction  with  complementary  Sea  Trial  field  experiments. 

(2)  Support  CD&E  through  representation  on  FCTs  assigned 
to  support  core  competency  areas. 

i .  Systems  Commands. 


(1)  Ensure  appropriate  R&D,  experimental  and  spiral 
development  activities  are  included  in  the  Sea  Trial  CD&E  Plan 
and  resource  participation  in  Sea  Trial  activities  as  required. 

(2)  Support  Sea  Trial  activity  in  coordination  with 

NWDC  . 
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(3)  Provide  representation  to  FCTs  as  assigned. 

(4)  Provide  input  to  CFFC,  in  conjunction  with  OPNAV,  on 
the  capability  of  the  industrial  base  to  evaluate  the  concepts 
under  development  as  part  of  the  Sea  Trial  efforts  with 
particular  focus  in  meeting  MCP  identified  gaps. 

(5)  Provide  representation  to  the  STESG . 

j .  CG  MCCDC  (DC,  CD) ■ 

(1)  Serve  as  a  supporting  commander  as  the  USMC  Sea 
Trial  Coordinator  and  the  supported  commander  for  USMC /Navy  Sea 
Trial  CD&E  activities. 

(2)  Coordinate  USMC  CD&E  activities,  including 
participation  in  FORCEnet,  as  required  to  support  the  Sea  Trial 
CD&E  Plan. 

(3)  Coordinate  USMC  Training  &  Education  Command 
(TECOM) ,  efdc  and  MCWL  activities  to  include  wargaming  as 
required  to  support  Sea  Trial  CD&E  Plan. 

(4)  Coordinate  USMC  S&T  efforts  and  Marine  Corps  Systems 
Command  (MCSC),  HQMC  and  Marine  Corps  Advocate,  and  Joint  CD&E 
activities  in  support  of  Sea  Trial  CD&E  Plan. 

(5>  Coordinate  assignment  of  USMC  Operational  Forces,  as 
fleet  counterparts,  to  participate  as  Sea  Trial  Operational 
Agents,  and  assign  USMC  representatives  to  the  FCTs. 

(6>  Provide  representation  to  the  STESG. 

(7)  Provide  input  to  STESG  recommendations. 

k.  TYCOMS  Provide  representation  to  FCTs  as  assigned. 

l .  Naval  Postgraduate  School . 


(1)  Support  Sea  Trial  through  conceptual  development 
and  specific  research  in  the  areas  of  Sea  Shield,  Sea  Strike, 
Sea  Basing  and  FORCEnet  in  accordance  with  CFFC/NPS  Memorandum 
of  Agreement. 

(2)  In  conjunction  with  NWDC,  support  analysis  of  Sea 
Trial  Event  products.  Additionally,  develop  and  support  long 
term  Red  Team  Cells  committed  :fcp  representing  potential  major 
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conflict,  terrorism  and  other  threats.  Red  Cells  will  be  tasked 
to  support  war  games  and  other  experiments  in  order  to  raise  the 
level  of  Red  Team  play,  and  to  represent  the  potential 
capabilities  joint  forces  may  have  to  overcome  in  the  future. 


m.  COMOPTEVFOR .  is  a  supporting  commander  for  planning, 
evaluation  and  operational  impact  assessment  of  Sea  Trial 


initiatives . 


Distribution : 

SNDL  Parts  1  and  2 
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START 


RESTART 
(Update  STIMS) 


CD&E  Initiatives 


START/ENTER  Process  via 
STIMS  database.  Events 
submitted  by  FCTs,  OPNAV, 
ONR,  Industry,  SYSCOMS, 
other  agencies, 

NWDC  distribute  to  ali 
stakeholders. 


Rejected 


OA  inform  sponsor/ 
CFFC  of  reason  for 
rejection. 


OA  VALIDATION  X 

*  Technically 

fe  asi  b  I  e/re  levant? 

*  Relevant  to  the 
CD&E  Plan? 

*  Meet  Fleet  priority? 

*  Meet  Mission 
Capability  Gap? 

*  Appropri  ate  venue 
identified? 

*  Etc.  / 


•  OA  conducts  validation 
assisted  by  NWDC  and 
FCT. 


NWDC  work  to  resolve 
or  brief  to  STESG  as 
candidate  w/o  resources. 


Resources 

Available? 


Sponsor  develops 
Experiment  Plan  and  Data 
Collection  &  Analysis  Plan 
NWDC  aligns  events 
across  all  pillars  and 
presents  consolidated 
Execution  Plan  to  STESG 
CFFC  designates  Military 
Utility  Assessment  (MU  A) 
Team  and  timeline 

o  Proposed  by  OA 


Plan,  develop,  and  coordinate  Sea  Trial  Event 


Conduct  Event 


Military  Utility 
Assessment 


OA  publish  Quick  look 
Event  sponsor  complete 
analysis 

MU  A  team  completes 
MU  A  and  submits  to  OA 
for  approval. 
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\ 


Significant  Enhanced  Warfighting  Capability 
Impact. 


Enhanced  Warfighting  Capability  Impact. 


CFFC  approve/disapprove  and  deliver  results 
to  N70  for  entry  into  Navy  Staff 


N80/N82  (NWDC 
assist) 

NWDC  track  & 
update  data  base 


17 
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